la ritual de exploración

Please don’t teach your customers to trust unknown numbers

2022-04-14T02:00:00+00:00

Teaching customers to trust cold-calls from random mobile numbers is not something we should be doing these days….

A few days ago I had a missed call from a mobile number I didn’t recognise. The caller did not leave a voicemail.

So, I called it back. I usually would not, but looking at reverseaustralia.com I could see that it may be my health insurance provider. The number is not listed anywhere that I could find on their website.

*“Hi, you’ve reached …. could I please have your policy number”*

At which point I asked how I can verify they’re who they say they are, and advised the general issue I had with this situation. They basically just insisted they are who they say they are… so I said I’ll call back on the listed number on the website.

Which I did, and sure enough it was actually the company who had called me. I asked about this mobile number and said my piece about how this feels dodgy… and was advised this is actually a ‘security feature’ used by their ‘backend team’.

They apparently use a system of cycling mobile numbers for outbound calls, and there’s apparently no process in place of how to help customers verify the caller, and nothing on the website about this practice. There’s also little information for their callcenter on what to say to customers asking about it as he was grasping at straws somewhat trying to answer my questions - “It is secure because they ask people security verification questions when they call them.” - which is the problem here!

There has been a growing awareness of vishing, and many organisations are trying to educate people to this threat. If you’re recieving a call from an unverified caller, don’t trust it. Teaching your customers to trust cold-calls from random numbers, asking for your policy details and your security verification answers is poor practice.

I feel like a much better approach would involve:

Calls come from a number listed on the website.
Leave a voicemail, advising to call a listed number for the company.
Write about the process on the website.
Give callcenter staff a playbook and info about how privacy is ensured during outbound calls.
I believe there’s a method for companies to have their numbers show up as text, I’m not much of a telco expert but surely it’s available if you’re a huge organisation such as this.

Do you have a story like this? I’m keen to hear in the comments about your experiences and other ideas of how companies could be making outbound calls in a much safer manner.

I’ve not embedded the company name in this post, I will likely disclose it in a week or after I have a response from the email I have sent them…. which isn’t a security email, just the general info email as that’s all they list on their site.

A Container Security Maturity Model

2022-04-11T02:00:00+00:00

Putting the CNCF Cloud Native Security Whitepaper into practice. An updated version of this post I published last month, now with the model in public Google Sheets instead of the repo as well as various edits.

This post is about building & using a maturity model to help assess & prioritise a subject area within cybersecurity. Specifically it is focused on drawing from the CNCF Cloud Native Security Whitepaper to measure the security posture of container workloads, from development through runtime. The process of making such a model should generally be useful outside of just container security.

Perhaps you’re a security engineer or an engineering lead. Cloud-first, shipping most things in containers. You may have more than a suspician that your container ecosystem lacks security features, that there are various risks and that there is a potential need to uplift. But it’s also a large subject area to address narrowly, there is a lot of complexity.

How can you effectively build a business case and determine your priorities within a large, complex set of systems and practices? I like to draw from a public model or framework, others in the industry have given input based on their experience and situations they have faced, and often showing a precedent of ‘industry practice’ will be of great help.

The resulting model is available in Google Sheets here. If anyone wants commenter access let me know.

Enter Capability Maturity Models

A Capability Maturity Model (CMM), or variation of such, is often employed when assessing software development processes. The general apprach is to have key process areas, each of which contain a set of related activities, and each activity has a set of criterea which are marked with a rating of 1-5, with 3 often being ‘good’ or ‘defined’. An example used within cybersecurity is Building Security in Maturity Model (BSIMM) - “an objective, data-driven evaluation that leaders seeking to improve their security postures can use to base decisions about resources, time, budget, and priorities”. It has a large community, but also complexity and a focus different to what we’re working on here. I want something ‘light touch’ and focused on the container ecosystem.

Capability Maturity Levels (from Wikipedea)

The Cloud Native Security Whitepaper

In late 2020 the Cloud Native Foundation TAG-Security published the Cloud Native Security Whitepaper. This defines & describes lifecycle Stages within a Cloud Native ecosystem - Develop, Distribute, Deploy, and Runtime as depicted below.

Coud Native Lifecycle

These lifecycle phases can be used as process areas in a model, grouping the processes & activities relating to container security.

There are other frameworks for looking at cloud & container ecosystems, such as the 4Cs, Redhat’s DevSecOps maturity model, and vendor models such as those seen on Aqua security homepage. As the CNCF is well placed as a vendor-neutral Cloud Native industry body, and their Security-TAG is extremely active and knowledgeable I felt that basing a model on the Cloud Native Security Whitepaper was most suitable.

Building the Model

Containers are not everything within cloud native, they’re somewhat intrinsic but there are various aspects of the lifecycle which may not be entirely relevant. Assessing an organisations entire cloud native approach is a much broader scope than I wanted or needed, as such various items have been omitted, or have been included but will get little direct attention from a container security program as they are already covered by other areas - an AppSec program (SCA, Secrets in Code, SAST), Cloud Security (eg AWS security policy adherance), SecOps (DFIR). The application sourcecode that will be hosted and the AWS account configuration are not covered in my scope. There will always be overlap or gaps, so I have tried to address these where appropriate.

The CNCF are also developing a Cloud Native Security Map which aims to map available tools, benchmarks and practices to the lifecycles.

The sections and rows in my model are:

Develop
- Standards
- Image Manifests
- Infrastructure as Code Manifests
Distribute
- Image Builds
- Image Registry Trust
- Image Registry Store
Deploy
- Image Trust
- Pre-flight Configuration Controls
- Pre-flight Policy Controls
Runtime
- Policy Enforcement
- Identity & Access Management
- Secure Storage
- Secrets Management
- Secure Network
- Observability & Incident Response

For ease of getting started, Excel was chosen, with a star chart displaying where we want to be (green) and where we are (red) based on a self-rating.

I spent a chunk of time thinking about each area and got it into something that I felt was workable. The definitions for each row are generally based upon the CNCF Security Whitepaper, and also largely informed by my individual experience of the capabilities, standards, ‘good practices’ in existence as well as what I’ve experienced over the years working in infosec. It’s imperfect and hopefully can grow and adjust over time.

Visualising Maturity

Utilising the Model

I rated each of the sections based on info I could find and what I knew, and got a nice star graph. Great! So, what to do with it…. put it on a billboard with “The end is near!” scrawled under it? No, let’s get a practical outcome. The measurement is there to help us drive a program of work, so let’s do that.

Based on the Whitepaper, the rows of the model and the technical and policy capabilities in the organistation I populated a Miro board to use in a group session. Each lifecycle phase is broken into sections, and in each we have our existing practices, and an area to note our gaps. There is also a box in each section for ‘research tasks’, items that may be noted but we’re not sure how useful it could be, which tech to go with etc.

Utilising Miro For Group Gap Analysis

Close Up of the Develop Lifecycle Stage within Miro

A session was held with our relevant security folks and most importantly with those who engineer container-based environments at an organisational practice level. This session was around 90 minutes where the model & lifecycle was introduced and then we went through the various stages giving people time to add notes. Before moving to the next section we’d quickly summarise and clarify, at the end of the session I got some overall feelings from people and gathered various themes, clarifying some of the items.

From this I went away and worked on a plan. We utilise Jira and so I took the lifecycle stages and themes identified within these as Epics, within each listing out the various gaps as Tasks or Stories - worded in such a way to highlight the remedial work and linked all together using a label plus another label for the lifecycle stage, eg ‘cncf-develop’. The various Epics were grouped under Initiatives. An example would be an Initative of “Implement & Enforce Security Capabilities In Image Builds”, under which one of the Epics of “Aggregate Image & Container Metadata into SIEM”, and within this one of the tasks of “Correlate Running Container IDs With Image Vulnerability Data”. Jira can be a bit clunky in organising complex pieces of work - these systems all are a bit - but it was helpful to get a view of the actual work. For more on structuring work within Jira there is a decent article from Atlassian here

Once it’s all in Jira it’s still just a big bunch of tasks. This is where the model helps to priopritise. We’d marked where we want to be, we’d marked where we are, and we have Jira labels allowing us to narrow into the areas. For me with starting such work I look at where we have the biggest gap, then which is the simplest to do. From here we know what is first up, and can reasonably plan for the next few months. Anything after that can be planned later. I won’t go into detail of what we picked - for you it might be implementing a break-build capability pre-push based on an agreed vulnerability threshold. It could be implementing an admission controller to allow policy adherence within your kubernetes stack. It all depends on your own outcome from the model.

Reflections

I like simple visual reprentations, based on some reasonable process to get there, and I’m happy with the result. It’s a bit rough, some of the language could be adjusted and more measurable metric definitions worked toward.

Having a published whitepaper from an industry body (the wonderful CNCF in this case) is incredibly valuable.

Having a published whitepaper from an industry body is incredibly valuable. Having the resultant visualisation of current and hoped maturity has been very effective in describing our problems and deciding where to increase resources.

Creating a model helped get my head around this complex space. Cloud Native and within that container security is entwined with all aspects of application & platform development and operations. Scoping helped to give a clearer picture of the container domain, and what was relevant within my organisation.I found that the Cloud Native Lifecycle Phases ends up with the largest proportion of items in Distribute & Runtime. This makes sense when you consider CI/CD often being the central place of earliest value stage gating (Distribute), and all of the moving pieces that could be involved in your production environments (Runtime). Once again, scoping can help here - especially with runtime - e.g EDR might be a pre-existing process handled by your SecOps team and wider than just cloud or containers, in a similar way you may already have standard Base OS hardening processes - so the container base host is out of scope.

It is hard to box all items into the phases - areas cross over, such as build capabilities (Distribute) being largely the same as IDE/local tooling in many situations (Develop). This is always the case with complex systems, trying to link these together in the task capture tool such as Jira or make a decision for the sake of progressing the work helps somewhat. The aim isn’t to neatly categorise things at the end of the day but to strengthen your security posture.

I used Miro for collaboration. People generally need to use it regularly to feel comfortable contributing. It is a great tool, the ability to visualise and all work together is especially important in remote work, but if your organisation suffers from a lack of standardisation in collaboration tool there will often be a barrier to effective contributions based on the comfort levels with the tool.

Jira is also hard - grouping things, prioritisation, visualising are all difficult at the best of times let alone if your workspace isn’t set up all that well. Once I had the uplift items in here though it was very easy to link to, show how long we’ve had these issues logged, drag items into a roadmap and collect relevant information.

The whitepaper & lifecycle are evolving, the CNCF Tag-Security is maintaining this great work. A new version of the whitepaper is in RFC now & the previously mentioned CNCF Security Map continues to collect open source tooling options. A space I’ll watch closely and hopefully get more involved in.

Image Vulnerability Scanners Compared, Pt1

2021-10-31T02:00:00+00:00

I’m looking at the capabilities and results from three container image vulnerability scanners - Trivy, Snyk & AWS ECR Scanning.

In this first segment I look at set up of the scans and gather vulnerability results for both a patched and unpatched image. In part 2 I will have a look at which appears to be more accurate in it’s findings, and other available features such as misconfiguration detection.

Introducing the tools

Trivy

Trivy is an open source vulnerability & misconfiguration scanner maintained by Aqua Security, and is very quick and easy to get started.

❯ trivy --version
Version: 0.20.2

Run with:

trivy <image> # for a table to stdout
trivy -q -f json <image> # for json output

A finding (in json) looks like this:

  {
    "VulnerabilityID": "CVE-2021-22945",
    "PkgName": "curl",
    "InstalledVersion": "7.78.0-r0",
    "FixedVersion": "7.79.0-r0",
    "Layer": {
      "DiffID": "sha256:a1920ef522ec8f49831ef40db2c3ad4777da04239270e6698d1674ebb90c8c82"
    },
    "SeveritySource": "nvd",
    "PrimaryURL": "https://avd.aquasec.com/nvd/cve-2021-22945",
    "Title": "curl: use-after-free and double-free in MQTT sending",
    "Description": "When sending data to an MQTT server, libcurl <= 7.73.0 and 7.78.0 could in some circumstances erroneously keep a pointer to an already freed memory area and bo
th use that again in a subsequent call to send data and also free it *again*.",
    "Severity": "CRITICAL",
    "CweIDs": [
      "CWE-415"
    ],
    "CVSS": {
      "nvd": {
        "V2Vector": "AV:N/AC:M/Au:N/C:P/I:N/A:P",
        "V3Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:H",
        "V2Score": 5.8,
        "V3Score": 9.1
      },
      "redhat": {
        "V3Vector": "CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:H",
        "V3Score": 7.4
      }
    },
    "References": [
      "https://curl.se/docs/CVE-2021-22945.html",
      "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-22945",
      "https://hackerone.com/reports/1269242",
      "https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/RWLEC6YVEM2HWUBX67SDGPSY4CQB72OE/",
      "https://security.netapp.com/advisory/ntap-20211029-0003/",
      "https://ubuntu.com/security/notices/USN-5079-1",
      "https://www.oracle.com/security-alerts/cpuoct2021.html"
    ],
    "PublishedDate": "2021-09-23T13:15:00Z",
    "LastModifiedDate": "2021-10-29T13:15:00Z"
  },

As well as vulnerability findings Trivy (in it’s json output) provides a lot of information about the image, including the build steps and layer history, image env, cmd, entry-pont, distro version etc. There’s quite a lot so I’m not pasting it here, but I do recommend having a look at this.

I will use the following jq filter to show succinct vulnerability information:

jq '.Results[].Vulnerabilities[] | {"ID": .VulnerabilityID, "name": .Title, "severity": .Severity, "pkgname": .PkgName, "installedversion": .InstalledVersion, "fixedversion": .FixedVersion}'

Pricing: Open Source / Free

Snyk

Snyk is a vulnerability scanner and SaaS. They provide a range of capabilities such as open source dependency scans, IaC and container scanning.

❯ snyk --version
1.745.0 (standalone)

Run with:

docker scan <image>

There is a json output, but I like the grepable output and bundling of vulnerabilities presented by default on the CLI for what I’m doing here. More information is available in json by passing in --json if desired.

A finding looks like:

✗ Critical severity vulnerability found in curl/libcurl
  Description: Double Free
  Info: https://snyk.io/vuln/SNYK-ALPINE313-CURL-1585246
  Introduced through: curl/libcurl@7.78.0-r0, curl/curl@7.78.0-r0
  From: curl/libcurl@7.78.0-r0
  From: curl/curl@7.78.0-r0 > curl/libcurl@7.78.0-r0
  From: curl/curl@7.78.0-r0
  Fixed in: 7.79.0-r0

Pricing: Free for individuals and small teams, this has some limitations in features available which don’t affect this CLI type of testing.

AWS ECR

AWS has an image scanning capability, which can be used to scan images within an AWS ECR repository. According to their docs “Each container image may be scanned once per 24 hours. Amazon ECR uses the Common Vulnerabilities and Exposures (CVEs) database from the open-source Clair project”

Homepage

Log into an ECR registry with the docker client. See the AWS Docs

Create a repository if you haven’t already. See the AWS Docs

ECR scans from the registry, so you need to push to the repository, set off a scan and then retrieve the results:

docker tag nginxlab:latest account.dkr.ecr.region.amazonaws.com/nginxlab:latest
docker push account.dkr.ecr.region.amazonaws.com/nginxlab:latest
aws ecr start-image-scan --repository-name nginxlab --image-id imageTag=latest
aws ecr describe-image-scan-findings --repository-name nginxlab --image-id imageTag=latest

A finding looks like this:

{
  "name": "CVE-2021-22947",
  "uri": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-22947",
  "severity": "MEDIUM",
  "attributes": [
    {
      "key": "package_version",
      "value": "7.78.0-r0"
    },
    {
      "key": "package_name",
      "value": "curl"
    },
    {
      "key": "CVSS2_VECTOR",
      "value": "AV:N/AC:M/Au:N/C:N/I:P/A:N"
    },
    {
      "key": "CVSS2_SCORE",
      "value": "4.3"
    }
  ]
},

…feels a bit sparse to me.

Summary information is also presented:

    "imageScanCompletedAt": "2021-11-01T11:58:01+11:00",
    "vulnerabilitySourceUpdatedAt": "2021-11-01T03:11:25+11:00",
    "findingSeverityCounts": {
      "MEDIUM": 3,
      "LOW": 2
    }
  },
  "registryId": "1234567890",
  "repositoryName": "nginxlab",
  "imageId": {
    "imageDigest": "sha256:b6e9f2d1354314363eb6c5afce9a186214afe2c5f1767dbc3397b1c87bcb3745",
    "imageTag": "latest"
  },
  "imageScanStatus": {
    "status": "COMPLETE",
    "description": "The scan was completed successfully."
  }
}

Pricing: From what I can tell ECR vulnerability scanning doesn’t incur a charge, but I can’t find any docs on this. The ECR Pricing doc suggests fees are only for the amount of data stored or transferred. Probably for a small self-hosted project this might be a few dollars per month. If you’re an organisation running builds thousands of times per day and have to upload each image to ECR before a vuln scan step then it could start to add up.

I will be using the following jq filter to present a succinct view of the findings from ECR:

jq '.imageScanFindings.findings[] | {"id": .name,"severity": .severity,"pkgname": .attributes[1].value,"installedversion":.attributes[0].value}'

First Image - Nginx Alpine

Running a basic health check endpoint and no configurations altered.

Unpatched

No OS upgrades applied via APK.

FROM nginx:1.20-alpine
RUN echo 'server {location /health {default_type text/plain; return 200 "OK";}}' \
    > /etc/nginx/conf.d/default.conf

Built with docker build . -t nginxlab:latest

Trivy

2 Critical, 2 High, 3 Medium

{
  "id": "CVE-2021-22945",
  "name": "curl: use-after-free and double-free in MQTT sending",
  "severity": "CRITICAL",
  "pkgname": "curl",
  "installedversion": "7.78.0-r0",
  "fixedversion": "7.79.0-r0"
}
{
  "id": "CVE-2021-22946",
  "name": "curl: Requirement to use TLS not properly enforced for IMAP, POP3, and FTP protocols",
  "severity": "HIGH",
  "pkgname": "curl",
  "installedversion": "7.78.0-r0",
  "fixedversion": "7.79.0-r0"
}
{
  "id": "CVE-2021-22947",
  "name": "curl: Server responses received before STARTTLS processed after TLS handshake",
  "severity": "MEDIUM",
  "pkgname": "curl",
  "installedversion": "7.78.0-r0",
  "fixedversion": "7.79.0-r0"
}
{
  "id": "CVE-2021-22945",
  "name": "curl: use-after-free and double-free in MQTT sending",
  "severity": "CRITICAL",
  "pkgname": "libcurl",
  "installedversion": "7.78.0-r0",
  "fixedversion": "7.79.0-r0"
}
{
  "id": "CVE-2021-22946",
  "name": "curl: Requirement to use TLS not properly enforced for IMAP, POP3, and FTP protocols",
  "severity": "HIGH",
  "pkgname": "libcurl",
  "installedversion": "7.78.0-r0",
  "fixedversion": "7.79.0-r0"
}
{
  "id": "CVE-2021-22947",
  "name": "curl: Server responses received before STARTTLS processed after TLS handshake",
  "severity": "MEDIUM",
  "pkgname": "libcurl",
  "installedversion": "7.78.0-r0",
  "fixedversion": "7.79.0-r0"
}
{
  "id": "CVE-2021-40528",
  "name": "libgcrypt: ElGamal implementation allows plaintext recovery",
  "severity": "MEDIUM",
  "pkgname": "libgcrypt",
  "installedversion": "1.8.8-r0",
  "fixedversion": "1.8.8-r1"
}

Snyk

1 Critical, 1 High, 2 Medium

✗ Medium severity vulnerability found in libgcrypt/libgcrypt
  Description: Use of a Broken or Risky Cryptographic Algorithm
  Info: https://snyk.io/vuln/SNYK-ALPINE313-LIBGCRYPT-1585259
  Introduced through: libgcrypt/libgcrypt@1.8.8-r0, libxslt/libxslt@1.1.34-r0
  From: libgcrypt/libgcrypt@1.8.8-r0
  From: libxslt/libxslt@1.1.34-r0 > libgcrypt/libgcrypt@1.8.8-r0
  Fixed in: 1.8.8-r1

✗ Medium severity vulnerability found in curl/libcurl
  Description: Insufficient Verification of Data Authenticity
  Info: https://snyk.io/vuln/SNYK-ALPINE313-CURL-1585257
  Introduced through: curl/libcurl@7.78.0-r0, curl/curl@7.78.0-r0
  From: curl/libcurl@7.78.0-r0
  From: curl/curl@7.78.0-r0 > curl/libcurl@7.78.0-r0
  From: curl/curl@7.78.0-r0
  Fixed in: 7.79.0-r0

✗ High severity vulnerability found in curl/libcurl
  Description: Cleartext Transmission of Sensitive Information
  Info: https://snyk.io/vuln/SNYK-ALPINE313-CURL-1585248
  Introduced through: curl/libcurl@7.78.0-r0, curl/curl@7.78.0-r0
  From: curl/libcurl@7.78.0-r0
  From: curl/curl@7.78.0-r0 > curl/libcurl@7.78.0-r0
  From: curl/curl@7.78.0-r0
  Fixed in: 7.79.0-r0

✗ Critical severity vulnerability found in curl/libcurl
  Description: Double Free
  Info: https://snyk.io/vuln/SNYK-ALPINE313-CURL-1585246
  Introduced through: curl/libcurl@7.78.0-r0, curl/curl@7.78.0-r0
  From: curl/libcurl@7.78.0-r0
  From: curl/curl@7.78.0-r0 > curl/libcurl@7.78.0-r0
  From: curl/curl@7.78.0-r0
  Fixed in: 7.79.0-r0

AWS ECR

3 Medium, 2 low

{
  "id": "CVE-2021-22947",
  "severity": "MEDIUM",
  "pkgname": "curl",
  "installedversion": "7.78.0-r0"
}
{
  "id": "CVE-2021-22945",
  "severity": "MEDIUM",
  "pkgname": "curl",
  "installedversion": "7.78.0-r0"
}
{
  "id": "CVE-2021-22946",
  "severity": "MEDIUM",
  "pkgname": "curl",
  "installedversion": "7.78.0-r0"
}
{
  "id": "CVE-2021-40528",
  "severity": "LOW",
  "pkgname": "libgcrypt",
  "installedversion": "1.8.8-r0"
}
{
  "id": "CVE-2020-28928",
  "severity": "LOW",
  "pkgname": "musl",
  "installedversion": "1.2.2-r1"
}

Patched

This time upgrading via the distributions package manager.

FROM nginx:1.20-alpine
RUN apk update && apk upgrade
RUN echo 'server {location /health {default_type text/plain; return 200 "OK";}}' \
    > /etc/nginx/conf.d/default.conf

Trivy

No vulnerabilities detected.

Snyk

No vulnerabilities detected.

AWS ECR

AWS detected one Low severity vulnerability.

{
  "id": "CVE-2020-28928",
  "severity": "LOW",
  "pkgname": "musl",
  "installedversion": "1.2.2-r1"
}

… but was it accurate?

The package version matches that which is reported:

❯ docker run -it --rm nginxlab:latest apk list musl
musl-1.2.2-r1 x86_64 {musl} (MIT) [installed]

But looking at the NVD record for CVE-2020-28928 this version appears to not be affected:

In musl libc through 1.2.1, wcsnrtombs mishandles particular combinations of destination buffer size and source character limit

So we have a false positive here due to an inaccurate match between the affected versions and that installed.

Part 2

Part 2 coming soon, with a deeper look at the results of from the unpatched image and a test of further scan features such as detection of misconfigurations.

Yamaha MT4X Multitrack Cassette Recorder: Disassembly & Service

2020-11-29T02:00:00+00:00

I’ve recently serviced a couple of Yamaha mt4x four track cassette recorders. Here I overview the process that I took and try to share info that I couldn’t easily find online myself.

The images used are mostly from the restoration of a 100V machine I picked up on eBay. When I got it it wouldn’t power on and was very dirty inside and out. I have added images from another (240v) machine in some places to show differences or where I missed photographing something originally.

There are probably some approaches that could be improved or things to know, so if you have any experience with these I’d be keen to hear your experience.

Starting Point: Dead & Dirty MT4X

Gather Tools & Hardware

Before unscrewing anything, you’ll want to make sure you’ve got a few things ready. I make sure I have the following available.

2x Phillips Screwdrivers. A little one, and a bigger one.
Tweezers. I prefer to have both a bent and straight set on hand, but one will do.
Needle Nose Pliers.
A marker or two.
Plastic pry Tool. I have a cheap phone openining tool, but a public transport card usually works too.
Cotton/Q Tips. Best to have quite a few on hand.
Vernier Calipers are an optional which can help with measuring belts if you don’t know exactly what you need or have.
A toothbrush and/or stiff brush for cleaning plastic parts.
Electronic cleaning & lubricating solutions. I use:
- a solvent spray for jacks.
- ‘clean & lubricant’ with a spray nozzle adapter for pots & switches.
- oil for motors & mechanical parts. Often I’ll use a sewing machine or motor oil but here am trying out inox mx3.
- Isopropyl.
Something to keep screws organised. Small plastic pots, jar lids etc are good options.
A toothpick or similar for applying oil.
A cassette tape for testing. A Chrome Type-II type is recommended.
A clear space to work. I like to utilise my standing desk and lay down a sheet of card - if someone wants to buy me an anti static mat, both xmas and my birthday are near at the time of writing this ;)
Time & patience! This stuff can get fiddly and frustrating, you want enough time to not rush things, and space to be able to leave the unit disassembled and come back to it if you hit a snag.

I’m noting the ratings of fuses, servo and belts which are from my observations and measurements. I have been unable to find any values for these online and am yet to get hold of a service manual. I would greatly appreciate comments of official values, or your observed values if you are taking a unit apart.

1x 70mm x 1mm square belt
1x 115mm x 4.7mm flat belt
1x Sankyo SHU2L 12v motor
1x 240v 500mA 20mm fast blow fuse (100v model)
1x 240v 120mA 20mm fast blow fuse (240v model)

Gather your tools

Keeping fixings tidy saves a headache later on

Disassembly

Now that we have a space to work and appropriate tools for the job, let’s get into the fun and pull this thing apart.

The first thing you want to do - which I neglected to photograph - is remove all of the knobs. All knobs, and all mixer sliders will pull off. Some might be a bit sticky but just pull straight up and it will come off. Don’t try to remove the push buttons, they’re affixed from the inside.

Once the knobs are removed, flip the unit over. Be careful of the exposed knobs, you don’t want to break these (or scratch your desk). I find as long as you lay it down gently you’ll be fine, placing a clean rag (tea towels work well) first is helpful.

There are 10 screws to remove here. 7 long, 3 short.

Once those are out, flip it over with the rear facing you. Gently lift the top from the rear. You will find here a ground wire connected to the cassette mechanism. This needs to be removed before lifting the top further. Take the screw out and set it aside. A really nice thing about the mtx units is that the screws are generally all the same, so you can just keep all of these gold screws together.

Although usually it is obvious where grounding wires connect when putting the mt4x back together, this is where I tend to use a marker. I will usually draw a coloured line across the grounding connection before removal so that it is easy to find the connection point later.

Remove the mechanism grounding wire before opening fully

With that wire out of the way we can proceed to open the unit. Carefully pivit from the rear to open. There is another ground - this time a pin connector - that you’ll find starting to pull. Slip this off before opening further. You can choose to fully open the unit if you have space, or like me lean the top back against something. Always be careful not to pinch wires or apply too much pressure to connection points.

The open mt4x

Marking grounding points can prevent later confusion

With the unit open, lift the shielding layer from the PCB (no need to fully disconnect it), and remove the connections between the top and base. For the grey and blue ribbon connectors there is a black terminal block that can be pulled up to release. The white and red connectors can be pulled straight up. If something feels very stuck, double check what you’re doing before proceeding.

Disconnect the ribbon cables between from base circuit board

Clean, Lube & Replace Parts

With the unit top and base seperated, set the top aside somewhere safe, and we will work through servicing the base parts. First take some time to familiarise yourself with what you’ve got in front of you and look for any damage, fluff, signs of past water spills etc. Then before continuing, fit the shield back over the PCB. Have you spotted the fuse yet? It is not the most convenient place, hidden away under the cassette mechanism.

mt4x base guts exposed

Lay the shielding back before continuing work

mt4x fuse is located under the cassette mechanism

A view of the mt4x heads

With a small brush and tweezers remove any dust and fluff from the cassette mechanism, taking care not to touch the heads at this stage. Once you’ve done this, remove the three remaining screws holding the mechanism in place.

Remove the three corner screws from the cassette machanism

Gently flip the mechanism over to the right, resting it upon the shielding.

From here we have access to the fuse. If it needs replacing, do so now. What a pain to replace a fuse huh!

Fuse in a 100v model

Fuse in a 240v model

Once again, clean out the exposed area, taking care of any electronic parts.

Now, there are four small screws holding the cassette mech cover in place. Remove these, carefully setting them aside and ensuring you don’t lose the two plastic latche parts. With this removed you will be able to remove the belts. If they have deteriorated you may need to clean their path in the wheels with a little warm soapy water on a q-tip.

To oil the servo we need to remove it. There are three screws for this, and once it is out you’ll want to pull the plastic belt wheel off. Give this a clean, and then oil the motor. To oil the motor use a toothpick or similar device rather than trying to apply direct from an oil dispenser. Give a couple of drops into the shaft and then gently rotate it a couple of times. Clean any residue from the exposed shaft before pushing the belt wheel back on, leaving a slight gap between it and the motor body.

Refix the motor, and then replace the belts (ideally with new ones). Refix the mech cover, being careful not to break the plastic parts.

Back of cassette mechanism

Remove these small screws to expose the belts

The mt4x belts removed

Remove these screws to remove the motor

Oil the shaft

Leave a slight gap when refitting

Refix the cassette machnism in place, and now we will clean the tape heads. Pour a little isopropyl alcohol into a canister, dipping a cotton swab and then using this to gently wide across the heads. If any dirt is visible on the swab tip, flip it over and use the other end until it is clean. I also clean the steel mechanical parts with iso, and the pinch rollers with a little warm, slightly soapy water.

Now just make sure you’ve cleaned out al the dust and fluff from the base before we start on the top section.

Isopropyl and cotton swab for head cleaning

Cleaned cassette mechanism and heads

A cheap paintbrush is a great tool for cleaning

Set aside the base in a safe place, and pop the top section down in your work space, face down. Lift off the protective shield, but don’t disconnect it. We’re now going to remove all of the PCBs and empty out this section to give it all a good clean.

Top section of the unit, ready to be worked on

There are plastic pins holding in the 1/4” input and RCA jacks. These are two on the inside, and four at the rear of the unit. For those on the rear, there are some you can get to from the inside with a small flathead screwdriver - use this to push them out a bit. From here, and for the rest, I use a plastic opening tool to gently pry these out. Be careful here not to break anything, and pull off the rear plate.

Push the rear plastic pins out a bit using a small screwdriver

Outer pin removal

Inner pin removal

One of the inner pins after removal

With the pins removed, pull this section out. It can be a little tricky, just work it until you find the right angle, and don’t force it. Once you’ve got it loose, disconnect the two ribbon cables. To do this, pull up the plastic outer to loosen and then lift the ribbons straight up.

Loosen connectors and then lift ribbons straight out

There is now a series of screws to remove to get these PCBs out. As you do so, gently remove any cables which connect things. Generally these are self-evident as to where they go when reconnecting, but you can always mark connections with a marker as we’ve done earlier. Some screws may be covered by tape. Place all PCBs in a safe place, an antistatic mat is a good choice but I generally just use a wooden dining table.

Three PCBs to remove, as shown here

Now we have an empty shell! There is card lining, leave it in. Pull out the button matrix. Give it all a brush out to remove any fluff etc.

Empty top shell

Button matrix

From here I take it to the bathroom and mix up a small amount of sugar soap with warm water. Using an old toothbrush and a cloth work your way around the unit. Don’t run it under water or get excessive wetness in inside due to the card. Once it’s all pretty clean run over it with a cloth and fresh clean water, then set aside to dry.

We now turn our attention to the circuit boards, with their switches and pots. My process is to first remove any large amounts of fluff etc by hand and tweezer. Then use an air can to blow off any loose dust. From here, I use a spray Electrical Clean and Lube with applicator nozzle to each pot and slider, one at a time. Spray, move through a full motion, repeat for each one. As you go through, you’ll find from turning and applying spray that more grime is able to be picked up so remove this as you go.

For the jacks I spray in some electronic cleaner (not lube) and gently insert a cotton swab to apply some friction to the contacts. Cotton swabs are probably not the best as they can leave cotton behind, but if you’re careful you can avoid this.

This can be fairly time consuming, just be slow and methodical about it. The results are worth it.

Reassembly

At this stage you will have clean boards, and a clean and dry shell. Now reverse the steps taken to remove everything to put it back together. If ribbon connector pins are a little bent, straighten these before slotting back in. When it comes to the plastic pins squeeze the ends together with some small pliers to insert back into their holes.

Squeeze pin ends to insert

Attach the base to the top, following the order of removal and any markings you made for connectors. When joining the top and botton ensure no cables are pinched.

Once it’s all together, and before screwing the top and base together or replacing all the knobs, test the unit. Use a guitar, bass or whatever you have at hand with a 1/4” cable and record something to each channel. Then listen back, and rotate all knobs and move sliders. I use headphones. Some of the jacks I haven’t tested at this time such as tape out, but they worked before so I’m fairly happy to test them in the studio later.

Testing the unit

Once you’re happy with it, reattach the knobs and screw the unit together. If all goes well you’ll have a shiny well functioning unit.

Hit Record

I hope this has been helpful to someone. There are various items I haven’t covered, which I hopefully will at a later date, such as demagnetising, motor speed adjustment and azumith adjustment, but these are often not necessary and a bit more advanced or require more expensive tools.

I’d love to hear from others with part numbers for these units, and any corrections, questions and ideas in the comments.

The clean, finished unit

Mapping the 12 Factor App to Boehm’s Quality Model

2020-08-06T14:21:00+00:00

I’ve recently been thinking about the methods of developing computer programs, and have been reading up on some work related to this.

I quite liked Boehm’s Quality Model, and thought it’d be interesting to map the 12 Factor App methodology to Boehm’s 7 Quality Factors.

I expect I will continue to grow this page over time. The plantuml for this diagram is here

Python set comparisons

2020-06-16T07:15:00+00:00

Today a colleague posted the following code snippet:

actual = set(["foo", "bar"] + [<other strings here>])  
expected = set(["bar", "foo"] + [<other strings here])
if not actual == expected: 
    print(f"Unexpected: {actual - expected}")

They were facing the occassional error of Unexpected: set() and felt a bit lost.

After originally thinking about the ordering and edge cases that could affect it somehow to cause the match to be True but also reduce to zero, I came to look at the subtraction and realised that this would happen when when all elements of actual exist within expected, but expected also has additional values.

For example:

>>> actual = set(["bar", "foo"])
>>> expected = set(['foo', 'bar', 'car'])
>>> actual == expected
False
>>> actual - expected
set()

Whereas if that was reversed, you get the additional value within actual, as it was never subtracted away:

>>> actual = set(['foo', 'bar', 'car'])
>>> expected = set(["bar", "foo"])
>>> actual == expected
False
>>> actual - expected
{'car'}

When working with sets, subtraction may not always give you what you think, and so methods have been provided for comparisons. In this case, python provides .difference() and .symmetric_difference():

symmetric_difference() would be fitting, as it will emit any item which is not common across both sets. In fact, this is essentially an XOR operation, and python does provide an operator for this - ^

And so, we could rewrite the comparison with the method:

>>> actual.symmetric_difference(expected)
{'car'}

or, using the operator:

>>> actual ^ expected
{'car'}

We’d then need to identify if the element was in actual or expected, but I’ll leave it as an exercise for the reader.

grep mutliple patterns

2020-04-15T08:15:00+00:00

We all have habits, and getting things done means that often we don’t look past them, or we’re not exposed to different ways of achieving something. Often I’ll sit back and think “actually let’s find another way of doing this”, or I see a cool trick and want to incororate it into my practice. Too often I don’t pass it on, or don’t practice it.

And so it goes with grep for me, where I’ve never really looking into multiple matches. So I spent a small chunk of time to work it out.

Matching multiple patterns using egrep

I had just done one of these:

$ kubectl describe nodes|grep Taints
Taints:             <none>
Taints:             <none>
$ kubectl describe nodes|grep Labels
Labels:             beta.kubernetes.io/arch=amd64
Labels:             beta.kubernetes.io/arch=amd64

I can get the task done using this, and probably in not many more keystrokes, but what if I want these in a one-liner?

I checked man grep and didn’t see anything obvious, except of course regular expresions, supported when passing -E to grep, or otherwise just using egrep

so when can use:

egrep '(pattern1|pattern2|patternN)'

Going back to the original multi-grep we can do the following, and improve upon by also matching the Name field:

$ kubectl describe nodes | egrep '(^Name|aints|abels)'
Name:               k8smaster
Labels:             beta.kubernetes.io/arch=amd64
Taints:             <none>
Name:               k8sworker1
Labels:             beta.kubernetes.io/arch=amd64
Taints:             <none>`

note I realised after this that it’s a bad example in practice as thekubectl command could be returning multiple rows of labels if more than one is present, and we’ll only match one. The method of multiple greps is still sound however.

Ansible Vault key retrieval from Bitwarden

2020-03-15T00:10:00+00:00

I use Bitwarden as my password manager, and I highly recommend it. I also use Ansible a whole lot, and make heavy use of Ansible Vault.

I’ve recently started using Invoke as my orchestrator after crgm suggested it, which has got me thinking more about streamlining my processes. Today I hooked bitwarden into the mix. This is a first pass, but I’m happy with it.

First up, I installed the bitwarden cli. See their docs for install and login steps.

Once that is working and I found that I could retrieve a password, I configured my orchestration.

You can’t just print the key in the command arguments to ansible-playbook. The available options are to provide a password file (--vault-password-file), or to have ansible ask for the key to be typed (--ask-vault-pass). One could probably pipe the key in, but that feels pretty yucky to me, but not as yuck as writing cleartext creds to a file on disk. Fortunately --vault-password-file will also accept an executable that prints the key to stdout.

Steps

vault-pass.py:

#!/usr/bin/env python3

import subprocess

def get_vault_key(search='ansible vault key'):
    command = 'bw get password %s' % search
    key = subprocess.run(command.split(), check=True, stdout=subprocess.PIPE)
    return key.stdout.decode('utf-8')

print(get_vault_key())

It needs to be executable, so chmod u+x vault-pass.py

Deployment step in invokes tasks.py:

from invoke import task

@task
def deploy(c, verbose=False):
    options = ''
    if verbose:
        options += ' -vvv'
    c.run('ansible-playbook play.yml -i inventory --ask-become-pass --vault-password-file vault-pass.py' + options)

Now I can run invoke deploy and not have to type the vault key. I want to get rid of the become password next, but will be focusing on moving this approach toward a python module and ansible-runner before then.

nftables on Debian 10 with Docker and no iptables

2020-03-07T00:10:00+00:00

I run a Debian 10 system with no iptables, and some multi-container services via docker-compose. Dumping some notes here as it can be a real pain.

Installation

Install the userspace tools (the kernel modules are already baked in).

apt-get install nftables

To blacklist the iptables module, create /etc/modprobe.d/noiptables.conf with the following:

install ip_tables /bin/false
install ip6_tables /bin/false

Start and enable the service:

systemctl enable nftables.service
systemctl start nftables.service

Docker config

In /etc/docker/daemon.json:

{
  "iptables": false
}

Previously the flag –iptables=false was used as a systemd override, but this is no longer the case.

In /etc/systemd/system/docker.service.d/override.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --fixed-cidr=172.17.0.0/16

This sets a static range 172.17.0.0/16 for the docker network. If you haven’t used systemd overrides before, the blank ExecStart= is needed to clear the value before applying a new one.

Then we restart docker:

systemctl daemon-reload
systemctl restart docker

fuck you docker…. (docker-ce will always install iptables userspace tools as a dependency)

# apt-get remove iptables
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  aufs-dkms aufs-tools cgroupfs-mount dkms libltdl7 linux-headers-amd64
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  docker-ce iptables
0 upgraded, 0 newly installed, 2 to remove and 5 not upgraded.
After this operation, 112 MB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.

and so

ln -s /sbin/iptables-nft /local/sbin/iptables

nftables rules

A fairly simple config that works for me so far.

/etc/nftables.conf

#!/usr/sbin/nft -f

#whitelist IPs
define home = xxx.xxx.xxx.xxx
define work = xxx.xxx.xxx.xxx
#internal IPs
define docker_ipv4 = 172.17.0.0/16

flush ruleset

table inet filter {
  set whitelist_ips {
    type ipv4_addr
    flags constant, interval
    elements = { $home, $work }
  }
  chain base_checks {
    ct state {established, related} accept
    ct state invalid drop
  }
  chain input {
    type filter hook input priority 0; policy drop;
    jump base_checks
    iifname lo accept
    ip saddr $docker_ipv4 accept
    ip saddr != @whitelist_ips drop
    ip protocol icmp accept
    tcp dport ssh accept
    tcp dport https accept
    drop
  }
  chain forward {
    type filter hook forward priority 0; policy drop;
    ct state {established,related} accept
    ip saddr $docker_ipv4 oif eth0 accept
    ip saddr $docker_ipv4 oifname "br-*" accept
  }
  chain output {
    type filter hook output priority 0; policy accept;
  }
}
table ip nat {
  chain prerouting {
    type nat hook prerouting priority 0;
  }
  chain postrouting {
    type nat hook postrouting priority 100;
    ip saddr $docker_ipv4 oif eth0 masquerade
  }
}

This allows forwarding between docker interfaces and to eth0, NAT to the container network, and a filtered input.

Load up the new rules with systemctl reload nftables

Logging

nftables service logs can be viewed from journalctl --unit=nftables.service

Logging of rules can be set up such as:

ip saddr <someaddress> log prefix "Some Prefix: " accept

Reference Material

nftables wiki

Using Conftest to enforce Terraform allowed resources

2019-11-21T10:10:00+00:00

I’ve just been playing with conftest as a means to enforce a whitelist of allowed Terraform resources. Conftest “is a utility to help you write tests against structured configuration data” and uses the Open Policy Agent rego syntax.

The setup that I’m working with is a controlled CI environment, where agents run a docker build container which hosts the underlying build tools - such as terraform, aws cli etc. Makefiles are provided in this image also, which are included in a projects build. We use Cloudformation for deployment of AWS resources, and there is concern that people will start to use terraform for these, which we want to discourage and possibly prevent.

After playing for the afternoon I got to an OK point with this. There are of course issues - as the person running a build can insert abitrary commands in their build steps they could call terraform direct to bypass my current way of doing this - and so it is currently more of a helpful check than a strict policy enforcement, but I’m sure this could be addressed when I spend some more time on it.

In the build agent:

First install Terrraform and Conftest, and then set the following configuration.

/opt/policy/terraform/base.rego

package main

whitelist = [
  "datadog",
  "pagerduty"
]

deny[msg] {
  resources := input.resource_changes[_].type
  not check_resources(resources, whitelist)
  allowed := concat(", ", whitelist)
  msg = sprintf("Terraform plan will change prohibited resources which are not in the following namespaces: %v", [allowed])
}

# Checks whether the plan will cause resources with certain prefixes to change
check_resources(resource, allowed_prefixes) {
  startswith(resource, allowed_prefixes[_])
}

/opt/make/terraform.mk

tf-init: ##Init state and vars
	@terraform init

tf-plan: tf-init ##Validate and check for changes
	@terraform version && \
	terraform validate && \
	terraform plan -out /tmp/plan.tf

tf-conftest: tf-plan ##Check the validation with conftest
	@terraform show -json /tmp/plan.tf | conftest -p /opt/policy/terraform/ test -

tf-clean: ##Remove files created during make
	@rm -f /tmp/plan.tf

tf-apply: tf-init tf-plan tf-conftest tf-clean ##Apply changes
	@terraform apply -auto-approve

And now in a project:

Makefile

include /opt/make/terraform.mk

in your build steps

make tf-plan tf-conftest

#Release to prod
make tf-apply

Now when a build is run, conftest checks the terraform plan against it’s policy and if a resource has been defined which isn’t in the whitelist, the build will fail.

...
...
Terraform will perform the following actions:

  # aws_ebs_volume.example will be created
  + resource "aws_ebs_volume" "example" {
      + arn               = (known after apply)
      + availability_zone = "us-east-2a"
      + encrypted         = (known after apply)
      + id                = (known after apply)
      + iops              = (known after apply)
      + kms_key_id        = (known after apply)
      + size              = 10
      + snapshot_id       = (known after apply)
      + tags              = {
          + "Name" = "ConfTestTest"
        }
      + type              = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------


FAIL - Terraform plan will change prohibited resources which are not in the following namespaces: datadog, pagerduty
make: *** [/opt/make/terraform.mk:20: tf-conftest] Error 1

$ 