Find all instances of a matched pattern in a string in Python

To find those parts of a string which match a pattern specified using a Regular Expression, the following code is probably the simplest.

import re

source = "A string 123 with numbers 456 and letters 789"

pattern = "[0-9]+"

results = re.findall(pattern, source)

for match in results: 
   print(match)

Result:

123
456
789

All matches of the regex inside the source string will be stored in the results list.

 

Delete Element Inside an Array in JavaScript

To remove an element from the middle of an array at a given index, we use the splice function as below. For example, to delete at index 2:

const arr = [1, 2, 3, 4, 5]
arr.splice(2, 1)
console.log(arr)

Result:

[ 1, 2, 4, 5 ]

In general, to delete an element at a specific index:

const deleteAtIndex = (arr, index) => {
  arr.splice(index, 1)
}

arr = [1, 2, 3, 4, 5]
deleteAtIndex(arr, 2)
console.log(arr)

Result:

[ 1, 2, 4, 5 ]

 

The following are all options that one my think of that do not do what we want.

Using the delete operator leaves a gap inside the array:

const arr = [1, 2, 3, 4, 5]
delete arr[2]
console.log(arr)

Result:

[ 1, 2, <1 empty item>, 4, 5 ]

Attempting to set the element to null or undefined also does not remove an element; it merely sets the reference at that location to null or undefined:

const arr = [1, 2, 3, 4, 5]
arr[2] = undefined
console.log(arr)

Result:

[ 1, 2, undefined, 4, 5 ]

Prevent Committing Secrets with a Pre-Commit Hook

It is difficult to remove a secret once committed to a git repo; the following describes how to prevent a commit with a secret (password, key, etc) using a pre-commit hook.

Even when following good practices for managing secrets (such as Secrets-as-a-Service) there is still a risk of accidentally committing secrets to git repositories by mistake. For example, suppose we pull secrets to use locally from Vault or similar, write some code and run “git add –all”; a file not meant to be committed can end up in the commit. If that happens, and especially if the commit is pushed to a remote, we would need to take extensive measures to wipe the secret from git history (see references).

Therefore, it is ideal to prevent committing anything that looks like a secret or password to a repo in the first place. This is possible by looking for high Shannon entropy strings and other techniques.
The detect-secrets project from Yelp is good for this.
Note: we need Docker installed on the system to use this tool.

Install the Node wrapper for detect-secrets globally:

$ npm install -g detect-secrets

Suppose we have a simple JavaScript file like this in index.js:

const output = 'Hello world'

console.log(output)

Run the following to check files for potential secrets:

$ detect-secrets-launcher index.js

Checking file: index.js

No results show for this file.

Now we add a high entropy string that looks like a password or secret key:

const output = 'Hello world'
const pwd = 'i6nrcpoeqzo4vmd0wpsn7j1pdglwv56mxsjggqkczwa'

console.log(output)

Run the secret detection again; the report should show:

$ detect-secrets-launcher index.js

Checking file: index.js
Potential secrets about to be committed to git repo! Please rectify or
explicitly ignore with an inline `pragma: allowlist secret` comment.

Secret Type: Base64 High Entropy String
Location: index.js:2

This shows the tool detected the potential secret.

Note that the scanner needs the file to be known to git; the git add command made this true.

We can add the detect-secrets-launcher command to a git pre-commit hook to detect any attempt to commit secrets and fail the pre-commit hook if any secrets are detected. The command will set the exit status to 1 if any secrets are found.

See this post for instructions on adding a git hook with Husky.

This .huskyrc file will connect detect-secrets to the pre-commit hook:

{
  "hooks": {
    "pre-commit": "detect-secrets-launcher *.js"
  }
}

Now try to commit the index.js file with the secret included:

$ git add .
$ git commit -m "try to commit a secret"

husky > pre-commit (node v11.6.0)

Checking file: index.js
Potential secrets about to be committed to git repo! Please rectify or
explicitly ignore with an inline `pragma: allowlist secret` comment.

Secret Type: Base64 High Entropy String
Location: index.js:2

...

husky > pre-commit hook failed (add --no-verify to bypass)

For larger projects with  more files the command might be:

"pre-commit": "detect-secrets-launcher src/**/*.js"

This can be used in projects as a guard against developers committing secrets, by accident or otherwise.

References

https://github.com/Yelp/detect-secrets

https://www.npmjs.com/package/detect-secrets

https://help.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository

Add Search to Hyper

Hyper is a nice terminal emulator for MacOS and others which does not have a search feature by default.

To add search to the Hyper terminal emulator app:

Open ~/.hyper.js (or click Hyper/Preferences… ) with a text editor.

Find the plugins value and add the following:

...

plugins: [
  'hyper-search'
],

...

Then install the hyper search plugin:

$ npm install hyper-search

Now restart Hyper, and search should work the usual way (e.g. Command+f on MacOS).

 

References

https://www.npmjs.com/package/hyper-search

Add a Pre-Commit Git Hook to a Node.js Repo with Husky

We can easily add a pre-commit hook to a NodeJS repository using the Husky package. The example below shows how this works.

First install Husky:

$ npm install husky

Now add a file .huskyrc in the root of your repo with the JSON below:

{
  "hooks": {
    "pre-commit": "echo 'commit event detected'"
  }
}

We can test this out to prove it is working:

$ git add .
$ git commit -m "hook testing" 
husky > pre-commit (node v11.6.0)
commit event detected

The hook executed but the commit succeeds.

To demonstrate how the hook can fail and prevent commits, we can set the exit status code of the command:

{
  "hooks": {
    "pre-commit": "echo 'commit event detected'; exit 1"
  }
}

Now we will get the following result:

$ git commit -m "hook test"
husky > pre-commit (node v11.6.0)
commit event detected
husky > pre-commit hook failed (add --no-verify to bypass)

This shows that commits will not be allowed if the exit status code of the command in the pre-commit hook is 1 (error).

Naturally we want to do something useful with the pre-commit hook, e.g. making sure all unit tests pass: a command like Jest will set the exit status to 1 (error) if there failing tests. Assuming we have defined npm test with a command like Jest or similar we could set our hook to:

{
  "hooks": {
    "pre-commit": "npm test"
  }
}

Now committing code which fails unit tests will be impossible, unless we explicitly bypass the hook like this:

$ git commit -m "hook test" --no-verify

References

https://www.npmjs.com/package/husky

JavaScript ES6 const Keyword Allows Changes

The ES6 JavaScript const keyword prevents re-assignment of a reference, but it does not actually prevent data modification of any object being referenced.

Of course, a primitive value declared const cannot be changed:

> const pi = 3.1415926
> pi = 3
TypeError: Assignment to constant variable.

However, objects declared with const can still have their properties changed:

const myObject = {
  a: 5, 
  b: 10, 
  c: 15
} 

myObject.b = -1 // no error

console.dir(myObject)

Result:

{ a: 5, b: -1, c: 15 }

Similarly, arrays declared with const in ES6 can be edited! They simply cannot be re-assigned.

const arr = [1, 2, 3, 4, 5]

a[1] = -1 // this is fine

console.dir(arr)

Result:

[1, -1, 3, 4, 5]

Using methods to manipulate the const array is still allowed:

const arr = [1, 2, 3, 4, 5] 

arr.pop() // allowed

console.dir(arr)

Result:

[1, 2, 3, 4]

The following use of the delete operator is ok with const as well:

const arr = [10, 20, 30]

delete arr[1] // works

console.dir(arr)

Result:

[10, <1 empty item>, 30]

Note that delete leaves gaps in the array; for objects it will leave null.

Countless more examples are possible but the key point is: arrays and objects being referenced by a const variable can still be mutated.

Promise.all with async/await in JavaScript

Promise.all can be used to await an array of promises, all of which can resolve at any time individually. Imagine launching multiple simultaneous requests to many APIs for example, all with varying unpredictable response times.

Promise.all will only resolve once all of the promises in the input array have resolved.

The following example shows two functions which return in 1 and 2 seconds respectively. Note that we are using the sleep function from npm here for maximum simplicity. This can be installed with: npm install sleep.

const sleep = require('sleep')

const f = async () => {
  sleep.sleep(1) // 1 second
  console.log('f sleep done, returning...')
  return 'f result'
}

const g = async () => {
  sleep.sleep(2) // 2 seconds
  console.log('g sleep done, returning...')
  return 'g result'
}

(async () => {
  const promise1 = f()
  const promise2 = g()

  const allResults = await Promise.all( [promise1, promise2] )

  console.log('All results: ' + JSON.stringify(allResults))
})()

Note the IIFE because we cannot have a top-level async function using the async/await syntax.

If we don’t need the return values, we can simply do:

await Promise.all( [promise1, promise2] )

To capture only some of the results, we can use destructuring assignment:

const [ result1 ] = await Promise.all( [promise1, promise2] )

The combination of Promise.all and async/await provides a very readable and compact way to make many asynchronous calls and use the results.

 

Reduce verbose output and show only failing tests in Jest

Sometimes the output of running Jest unit tests is a bit too verbose, with the pass and fail statuses of all tests as well as the test coverage table.

To prevent outputting the code coverage report table, use:

jest --collectCoverage=false

(This assumes Jest is configured to show the code coverage report already.)

We can reduce the verbosity of the Jest output aside from the coverage table by using a custom reporter. Jest Silent Reporter is a good choice for this. To install it:

$ npm install jest-silent-reporter

Then run the tests with:

$ jest --reporters jest-silent-reporter

Now we will see only the failing test output.

The coverage report can still be shown after the silent reporter output.

So, for the minimum output possible we can combine the silent reporter and omit coverage report:

$ jest --collectCoverage=false --reporters jest-silent-reporter

We can add this command to our package.json to have the option to run a silent minimal output for all our tests easily, for example like this:

"scripts": {
  "unit:silent": "jest --reporters jest-silent-reporter --collectCoverage=false" 
}

Then we can just run:

$ npm run unit:silent

References

https://www.npmjs.com/package/jest-silent-reporter

 

Send email to a mailing list using the Mailgun API with cURL

Mailgun provides one of the nicest APIs for email communication.

This example assumes we have a mailing list already created and want to send an email to all recipients via the Mailgun API.

The way to do this is to POST to /messages and use the mailing list address as the recipient address in the API call. Here is the call being made via cURL:

$ curl --request POST \
 --url https://api:MY_API_KEY@api.mailgun.net/v3/mg.mydomain.com/messages \
 --form 'from=Name <noreply@mg.mydomain.com>' \
 --form "text=My email body" \
 --form "subject=My email subject line" \
 --form to=my-email-list@mg.mydomain.com \
 --form user=api:MY_API_KEY

Note that this API call uses form fields and not any JSON payload.

A successful response will look something like:

{
   "message": "Queued. Thank you.",
   "id": "..."
}

References

https://www.mailgun.com/email-api

Computed Property Names in JavaScript

Computed Property Names is an ES6 feature which allows the names of object properties in JavaScript object literal notation to be determined dynamically, i.e. computed.

JavaScript objects are really dictionaries, so it was always possible to dynamically create a string and use it as a key with the syntax object[‘property’] = value.

However, ES6 Computed Property Names allow us to use dynamically generated names within object literals. Example:

const myPropertyName = 'c'

const myObject = {
  a: 5,
  b: 10,
  [myPropertyName]: 15
} 

console.log(myObject.c) // prints 15

To stress that expressions can be used directly as computed property names, another example:

const fieldNumber = 3

const myObject = {
  field1: 5,
  field2: 10,
  ['field' + fieldNumber]: 15
}

console.log(myObject.field3) // prints 15

This can be very handy.

One more variation is to use template literals (string interpolation) for the computed property names — note that this still requires the square bracket syntax, however:

const fieldNumber = 3

const myObject = {
  field1: 5,
  field2: 10,
  [`field${fieldNumber}`]: 15
}

console.log(myObject.field3) // prints 15