failed to create runc console socket: mkdir /tmp/pty064803763: no space left on device: unknown

Got this error today when trying to log into a docker container

$ docker exec -it mysql /bin/bash
failed to create runc console socket: mkdir /tmp/pty064803763: no space left on device: unknown

Then I tried

$ df -h
Filesystem                 Size  Used Avail Use% Mounted on
devtmpfs                   3.9G     0  3.9G   0% /dev
tmpfs                      3.9G  4.0K  3.9G   1% /dev/shm
tmpfs                      3.9G  378M  3.6G  10% /run
tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv  7.8G  128M  7.3G   2% /
/dev/mapper/rootvg-usrlv   9.8G  2.3G  7.0G  25% /usr
/dev/sda1                  976M  113M  797M  13% /boot
/dev/mapper/rootvg-optlv   2.0G  1.5G  303M  84% /opt
/dev/mapper/rootvg-homelv  976M  3.9M  905M   1% /home
/dev/mapper/rootvg-tmplv   2.0G  432M  1.4G  24% /tmp
/dev/mapper/rootvg-varlv   7.8G  4.2G  3.3G  56% /var
/dev/sdc                   246G  1.1G  233G   1% /app
/dev/sdb1                   16G  2.1G   13G  14% /mnt/resource
tmpfs                      797M     0  797M   0% /run/user/48081

which did not indicate any problem. /tmp is only 24% used.

$ df -i
Filesystem                  Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                   1016516    431  1016085    1% /dev
tmpfs                      1019390      2  1019388    1% /dev/shm
tmpfs                      1019390    856  1018534    1% /run
tmpfs                      1019390     16  1019374    1% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv   524288   6703   517585    2% /
/dev/mapper/rootvg-usrlv    655360  86575   568785   14% /usr
/dev/sda1                    65536    343    65193    1% /boot
/dev/mapper/rootvg-optlv    131072  53934    77138   42% /opt
/dev/mapper/rootvg-homelv    65536    166    65370    1% /home
/dev/mapper/rootvg-tmplv    131072 131072        0  100% /tmp
/dev/mapper/rootvg-varlv    524288  26116   498172    5% /var
/dev/sdc                  16384000   3649 16380351    1% /app
/dev/sdb1                  1048576     13  1048563    1% /mnt/resource
tmpfs                      1019390      1  1019389    1% /run/user/48081

Above is showing all the inodes have been exhausted under /tmp. On listing /tmp I saw lot of files like this

#-> cat /tmp/tmp.01amFL2hH4
read_kt /etc/krb5.keytab
write_kt /tmp/tmp.kt

For now I manually deleted the files. And after that it works

$ docker exec -it mysql /bin/bash
root@48b2dea3706b:/home#
Posted in Software | Leave a comment

Understanding the internals of wordpress php-fpm-alpine image

This post describes the internals of wordpress:php7.4-fpm-alpine Docker image.

The wordpress container does nothing but run the following command and wait for it to end which it never does as it spins up a server that listens for incoming connections forever:

$ docker-entrypoint.sh php-fpm

The file docker-entrypoint.sh can be found in /usr/local/bin/docker-entrypoint.sh inside the container. It is instructive to read and understand this file. It will do some setup etc. such as installing wordpress files in the WORKDIR which defaults to /var/www/html and ends with

exec "$@"

what above line does is to run php-fpm as if it were a continuation of the docker-entrypoint.sh script [1].

php-fpm is PHP’s Fast CGI processor which spins up a server listening at port 9000.

By inspecting the docker image (docker image inspect wordpress:php7.4-fpm-alpine) one can see from where the script installs PHP. This is given by the PHP_URL in below:

"Env": [
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "PHPIZE_DEPS=autoconf \t\tdpkg-dev dpkg \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkgconf \t\tre2c",
    "PHP_INI_DIR=/usr/local/etc/php",
    "PHP_EXTRA_CONFIGURE_ARGS=--enable-fpm --with-fpm-user=www-data --with-fpm-group=www-data --disable-cgi",
    "PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64",
    "PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64",
    "PHP_LDFLAGS=-Wl,-O1 -pie",
    "GPG_KEYS=42670A7FE4D0441C8E4632349E4FDC074A4EF02D 5A52880781F755608BF815FC910DEB46F53EA312",
    "PHP_VERSION=7.4.5",
    "PHP_URL=https://www.php.net/get/php-7.4.5.tar.xz/from/this/mirror",
    "PHP_ASC_URL=https://www.php.net/get/php-7.4.5.tar.xz.asc/from/this/mirror",
    "PHP_SHA256=d059fd7f55bdc4d2eada15a00a2976697010d3631ef6f83149cc5289e1f23c2c",
    "PHP_MD5=",
    "WORDPRESS_VERSION=5.4",
    "WORDPRESS_SHA1=d5f1e6d7cadd72c11d086a2e1ede0a72f23d993e"
]

Also we can see from PHP_EXTRA_CONFIGURE_ARGS that the php-fpm process will run as www-data. There are also references to www-data in docker-entrypoint.sh.

TIP: If you want to change the directory in which wordpress gets installed, simply change the –workdir to the location in which you want wordpress to be installed.

WordPress Boot Sequence

There are two great articles explaining how wordpress loads up when a request is received:

The bad: it looks like this boot sequence happens on EVERY request. That is why you don’t need to refresh or restart anything if you make changes to wp-config.php for example.

The boot sequence is this. Under /var/www/html there is an index.php which loads wp-blog-header.php which loads wp-load.php which loads wp-config.php which loads wp-settings.php which loads as many as 36 files.

bash-5.0# grep -n wp-blog-header.php index.php
4: * wp-blog-header.php which does and tells WordPress to load the theme.
17:require __DIR__ . '/wp-blog-header.php';
bash-5.0# grep -n wp-load.php wp-blog-header.php
13:	require_once __DIR__ . '/wp-load.php';
bash-5.0# grep -n wp-config.php wp-load.php
4: * and loading the wp-config.php file. The wp-config.php
8: * If the wp-config.php file is not found then an error
10: * wp-config.php file.
12: * Will also search for wp-config.php in WordPress' parent
27: * If wp-config.php exists in the WordPress root, or if it exists in the root and wp-settings.php
28: * doesn't, load wp-config.php. The secondary check for wp-settings.php has the added benefit
34:if ( file_exists( ABSPATH . 'wp-config.php' ) ) {
37:	require_once ABSPATH . 'wp-config.php';
39:} elseif ( @file_exists( dirname( ABSPATH ) . '/wp-config.php' ) && ! @file_exists( dirname( ABSPATH ) . '/wp-settings.php' ) ) {
42:	require_once dirname( ABSPATH ) . '/wp-config.php';
76:		/* translators: %s: wp-config.php */
78:		'<code>wp-config.php</code>'
83:		__( 'https://wordpress.org/support/article/editing-wp-config-php/' )
86:		/* translators: %s: wp-config.php */
88:		'<code>wp-config.php</code>'
bash-5.0# grep -n wp-settings.php wp-config.php
108:require_once ABSPATH . 'wp-settings.php';

Request Flow

in my nginx conf, I have

location / {
    try_files $uri $uri/ /index.php?$args;
}

Now when a request comes in for https://mywordpresssite.com/2020/04/22/e2e-connected-visibility-platform/ nginx will first see if there is a resource with that name. There isn’t any so nginx will perform internal redirect and attempt to serve index.php?2020/04/22/e2e-connected-visibility-platform. This will kick-in the index.php under /var/www/htmland the wordpress boot sequence. So requests for pretty much all resources are going through index.php which acts as the Main method.

Posted in Software | Tagged | Leave a comment

Debugging File Upload Issues with WordPress

First make sure you have adequately increased the maximum file size that can be uploaded to wordpress. It is possible that uploads may still not work. On php-fpm-alpine image (and even on apache) wordpress runs under the credentials of www-data user. This user needs to have write access to the wp_upload_dir directory and the subdirectory under it where it will attempt to create the file. The full path resolves to /var/www/html/wp-content/uploads/2020/06 to give an example. Use the namei utility with the -om flag to see if www-data has sufficient permissions to be able to create a file in the directory. Use chown -R to make www-data the owner of all subdirectories under wp-content.

Posted in Software | Tagged | Leave a comment

How to change the maximum file size you can upload to wordpress on nginx w/ php-fpm

Apache is a solid web server but my goto environment for wordpress has now changed to running it on Docker with NGINX + php7.4-fpm-alpine. To change the max file size you can upload to wordpress, you will need to make changes to two places:

NGINX

In the server block set the client_max_body_size like so

server { 
    listen 443 ssl; 
    server_name '"$COMMON_NAME"'; 
    ssl_certificate /home/server.crt;
    ssl_certificate_key /home/server.key;
    root /var/www/html; 
    client_max_body_size 100M;
  

This setting controls how much the web server itself will allow before wordpress even comes into the picture.

php7.4-fpm-alpine

Create following file if it does not exist /usr/local/etc/php/conf.d/uploads.ini and edit it like so:

file_uploads = On
memory_limit = 500M
upload_max_filesize = 100M
post_max_size = 100M
max_execution_time = 600
max_file_uploads = 50000
max_execution_time = 5000
max_input_time = 5000

You are all set! Above settings control how much wordpress will allow. Take a look at wp_max_upload_size

Tip: Verify /usr/local/etc/php/conf.d/uploads.ini is indeed the correct file to modify by running phpinfo() in PHP REPL

bash-5.0# php -a
Interactive shell

php > phpinfo();
phpinfo()
PHP Version => 7.4.5

...

Configuration File (php.ini) Path => /usr/local/etc/php
Loaded Configuration File => (none)
Scan this dir for additional .ini files => /usr/local/etc/php/conf.d
Additional .ini files parsed => /usr/local/etc/php/conf.d/docker-php-ext-bcmath.ini,
/usr/local/etc/php/conf.d/docker-php-ext-exif.ini,
/usr/local/etc/php/conf.d/docker-php-ext-gd.ini,
/usr/local/etc/php/conf.d/docker-php-ext-imagick.ini,
/usr/local/etc/php/conf.d/docker-php-ext-mysqli.ini,
/usr/local/etc/php/conf.d/docker-php-ext-opcache.ini,
/usr/local/etc/php/conf.d/docker-php-ext-sodium.ini,
/usr/local/etc/php/conf.d/docker-php-ext-zip.ini,
/usr/local/etc/php/conf.d/error-logging.ini,
/usr/local/etc/php/conf.d/opcache-recommended.ini,
/usr/local/etc/php/conf.d/uploads.ini

Posted in Software | Tagged | Leave a comment

Rahul Gandhi with Narendra Modi in Restaurant

मोदी जी एक रेस्तरां में गये और राहुल के पास वाली सीट पर बैठ कर टी वी देखने लगे ।

टी वी पर रात 9 बजे वाली न्यूज चल रही थी जिसमें कोई आदमी बहुत ऊंची इमारत की छत से कूदकर आत्महत्या करने जा रहा था।

राहुल ने मोदी जी से पूछा,"आपको क्या लगता है, वो कूद जायेगा ?" मोदी जी ने कहा, "हां, वह कूद जायेगा, मैं शर्त लगा सकता हूं।"

राहुल बोला,"ठीक है, शर्त लगाई, वह नहीं कूदेगा।" ठीक है कहते हुए, मोदी जी ने दो हजार रुपए टेबल पर रख दिए।

जैसे ही राहुल ने अपने दो हजार रुपए टेबल पर रखे वह शख्स इमारत से नीचे कूद गया और मर गया। राहुल ने बड़ी निराशा और अनिच्छा से दो हजार रुपए मोदी जी को दे दिए।

मोदी जी बोले,"नहीं, मैं आपसे ये रुपए नहीं ले सकता क्योंकि मैने शाम पांच बजे की न्यूज में ही ये देख लिया था इस लिए मुझे पता था कि वह कूद जायेगा।"

राहुल बोला, "पांच बजे की न्यूज मे ये सब देखा तो मैने भी था, पर मैंने यह नहीं सोचा कि वह शख्स दोबारा ऐसा करेगा।"………..

अब मोदीजी ने पैसे ले लिए ।

😁😁🤣🤣😆😆

Posted in Jokes | Leave a comment

Complete guide to Azure AD Authentication in Node

There is a lot of documentation available on how to authenticate users against Azure AD. There are even libraries like MSAL.js that are supposed to do the job for you. Unfortunately, for me, I found the Microsoft documentation very noisy and confusing. Which library am I supposed to use? MSAL.js, ADAL.js, there is even a Passport.js in classic Microsoft fashion of developing multiple products for same use case – think Yammer, Teams, Skype or OneDrive, Sharepoint. Should I do OAuth or OpenIDConnect? It does not stop here. Microsoft provides two endpoints for doing OAuth – v1 and v2.

The link that is most helpful to understand and get the background on doing authentication using OAuth or OpenIdConnect (OIDC) is https://developer.okta.com/blog/2019/10/21/illustrated-guide-to-oauth-and-oidc. Start by reading this link and understand the four things you need in Azure before you can proceed further:

  • Client ID: identifier of your application. Also known as Application ID. You will get this by creating a new App Registration in Azure.
  • Tenant ID: identifier of the organization’s Azure AD against which you are trying to authenticate users.
  • Client Secret: password of your application. You create a new client secret in Azure portal.
  • Redirect URL: this has to be setup in Azure as well.

The OAuth authentication is a two step process. We use v1 endpoints in this article.

There are two endpoints involved in OAuth authentication. An authorization endpoint and a token endpoint.

1. In the first step, one makes a request to the authorization endpoint. A complete request might look like:

https://login.microsoftonline.com/109156be-c4fb-41ea-b1b4-efe1671c5836/oauth2/authorize/?client_id=9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d&response_type=code&response_mode=query&redirect_uri=https://foo.com/auth/callback&prompt=consent&state=https://foo.com/the/page/user/wanted/to/access

This endpoint will take the user to Microsoft login page. The user logs in. Then user is prompted to grant permissions to the app. If user approves, an access code is returned to the user’s browser and the browser is redirected to the redirect_uri – think of this as a callback url. The point to note here is that it is the user’s browser which will invoke the redirect_uri with the access code, not Azure. Azure does not call the callback. Next thing to note is that Azure only accepts https URL for the redirect_uri and this restriction comes from the protocol itself because of security considerations [https://tools.ietf.org/html/rfc6749#section-10.3]

An example callback might look like

https://foo.com/auth/callback?code=AQABAAIAAAAm-0gAA&state=%2f&session_state=790c56da-2615-484b-9200-54d08cb21482

Whatever you pass in the state parameter in the call to authorize, you will get it back in the callback. You should set the state parameter to the URL the user was trying to access in the first place. Otherwise, that information will be lost and you won’t be able to land the user to appropriate page after logging in the user.

2. In your callback method, you use the access code to get access token of the user. To do this you have to make a POST request to the token endpoint and you need to know a client secret. Node code for doing that would look like following

var postData = querystring.stringify({
                'grant_type': 'authorization_code',
                'code': req.query.code,
                'client_id': this.clientId,
                'client_secret': this.clientSecret,
                'resource': this.clientId,
                'redirect_uri': this.callbackUrl,
                'scope': 'user.read'
            });

var tokenRequestOptions = {
                hostname: this.tokenUrl.hostname,
                port: 443,
                path: this.tokenUrl.pathname,
                method: 'POST',
                timeout: 45,
                headers: {
                    'Content-Type': 'application/x-www-form-urlencoded',
                    'Content-Length': Buffer.byteLength(postData)
                }              				
            }
            
const tokenRequest = https.request(tokenRequestOptions, tokenResponse => {

A successful response has following fields in it and is returned as a string that you will have to parse using JSON.parse:

From here on, you can use either the access_token or the id_token to get user details. I found both have the same info. I used the id_token which is JWT token. To decode it, use the jwt-decode or jsonwebtoken library. The decoded token has following info in it

Decoded ID Token

The aud field is equal to your client ID and the UUID in iss field is equal to the issuer – the tenant ID. You can use this to validate the token. See this for what the other fields mean.

That’s it! You have authenticated the user! Observe that the client secret is never shared with the browser. The story does not end here because now you should set some persistent info – think cookie – in the response sent back to the browser so that when user visits the site again, they don’t have to log in again! Also, we need to redirect the browser to the page the user has been trying to visit all this time.

There are many ways you can set a cookie. I used following code where we use the jsonwebtoken library.

const userToken = {
            uid: userInfo.unique_name,
            name: userInfo.name
        };
        const signedToken = jwt.sign(userToken, this._jwtSecret, { expiresIn: 86400 }); // 24h
        const redirectUrl = url.resolve(this._siteUrl, landingUrl);
        res.status(200)
            .cookie(this._loginCookie, signedToken, { maxAge: 86400, httpOnly: true, secure: true })
            .redirect(redirectUrl);   

The secure=true option tells to transmit the cookie only via TLS and httpOnly=true prevents user from accessing the cookie via client-side javascript. Both are pretty important.

Now when user makes a request back again, s/he can be authenticated by examining the cookie. We use the cookie-parser library to access cookies using req.cookies in the code below.

const token = req && req.cookies && req.cookies[this._loginCookie];
        if (!token) {
            return false;
        }
        
        // Test the validity of the token
        try {
            const decodedToken = jwt.verify(token, this._jwtSecret);
            // Compare the token expiry (in seconds) to the current time (in milliseconds)
            // Bail out if the token has expired
            if (decodedToken.exp <= Date.now() / 1000) {
                return clearTokenAndNext();
            }            
            return {
                'email': decodedToken.uid,
                'name': decodedToken.name
            } 
        } catch (err) {
            return clearTokenAndNext();
        }

Finally we can wrap this up by adding a method that will logout a signed in user

logout(req, res) {
        this._clearLoginCookie(res);
        res.status(200).send('You have successfully logged out');
    }

    _clearLoginCookie(res) {
        res.clearCookie(this._loginCookie);        
    }

There you have it! With a little effort, it can all be done without using any library. You will need following dependencies:

"dependencies": {
    "cookie-parser": "~1.4.5",
    "express": "~4.17.1",
    "jwt-decode": "~2.2.0",
    "jsonwebtoken": "~8.5.1"
  }

Good Luck! Make it happen!

Implicit Grant explained: The best practice is to obtain the access or ID token on the server in the callback. But if you are developing a SPA which has no server-side code, then you have no choice but to obtain the token on the client side. To do this, you need to check the implicit grant boxes.

Again do this only if you have no server side code i.e., there is no choice but to get the token on client itself. https://tools.ietf.org/html/rfc6749#section-10.3 clearly states:

When using the implicit grant type, the access token is transmitted in the URI fragment, which can expose it to unauthorized parties.

If you enable implicit grant, you can acquire the access or ID token in the call to authorize itself as shown in https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-implicit-grant-flow

For this to work you need to use response_type=id_token instead of response_type=code when making request to the authorize endpoint. The implicit grant allows you to get the token directly from the authorize endpoint without having to know the client secret. It short-circuits the step of using the access code to get the access or ID token in the callback on the server. TL;DR: don’t do it.

V1 vs V2: Azure AD supports two endpoints for OAuth. What’s the difference?
https://nicolgit.github.io/AzureAD-Endopoint-V1-vs-V2-comparison/
https://docs.microsoft.com/en-us/azure/active-directory/azuread-dev/azure-ad-endpoint-comparison#openid-profile-and-email
scopes are only supported by the v2 endpoint. In my case the v1 endpoint was perfect as it returns all the information about the user. See https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-auth-code-flow and https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent for information on scopes.

  • The email scope allows your app access to the user’s primary email address through the email claim in the id_token, assuming the user has an addressable email address.
  • The profile scope affords your app access to all other basic information about the user, such as their name, preferred username, object ID, and so on, in the id_token.
Posted in Software | Tagged | Leave a comment

CloudFlare 522 Error. Server does not respond.

This is probably the longest time I have spent on debugging something (4 days) so its worth the effort to write about it.

The problem: We built a wordpress site but got CloudFlare 522 error when trying to connect to it. Our logs were completely empty.

The Setup: Our setup consisted of a VM in a private network with no public IP. This VM ran 3 docker containers connected together in a user-defined bridge network. The first container was an nginx server which would forward requests for PHP resources to the wordpress container. The third container was running MySQL 5.7. Nginx server configuration was borrowed from here. If the VM has no public IP how does one access it? The VM was connected to a layer-4 load balancer in a DMZ with a public IP. Further the load balancer would only accept traffic from CloudFlare CDN. The purpose of CloudFlare was to filter out malicious traffic and protect the servers.

CloudFlare -> Load Balancer -> NGINX -> WordPress -> MySQL

How we debugged: We tried all the usual things. We tested that the host is forwarding http (port 80) and https (port 443) traffic to the nginx container.

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                      NAMES
ec8600d9c4b4        nginx:1.17          "nginx -g 'daemon of…"   24 hours ago        Up 24 hours         0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   nginx
feb535607eb2        wordpress:php7.4-fpm-alpine       "docker-entrypoint.s…"   25 hours ago        Up 25 hours         9000/tcp                                   wordpress
48b2dea3706b        mysql:5.7           "docker-entrypoint.s…"   25 hours ago        Up 25 hours         0.0.0.0:3306->3306/tcp, 33060/tcp          mysql

We tested that wordpress container is listening on port 9000

$ docker exec wordpress netstat -tpln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.11:40662        0.0.0.0:*               LISTEN      -
tcp        0      0 :::9000                 :::*                    LISTEN      1/php-fpm.conf)

We tested that nginx container can connect to wordpress container

$ docker exec -it nginx /bin/bash
root@ec8600d9c4b4:/# apt-get install netcat
root@ec8600d9c4b4:/# nc -zv wordpress 9000
DNS fwd/rev mismatch: wordpress != wordpress.wordpress_net
wordpress [172.18.0.3] 9000 (?) open

We edit our wp-config.php to enable debug logs in wordpress

// Enable WP_DEBUG mode
define( 'WP_DEBUG', true );

// Enable Debug logging to the /wp-content/debug.log file
define( 'WP_DEBUG_LOG', true );

// Disable display of errors and warnings
define( 'WP_DEBUG_DISPLAY', false );
@ini_set( 'display_errors', 0 );

// Use dev versions of core JS and CSS files (only needed if you are modifying these core files)
define( 'SCRIPT_DEBUG', true );

We then tailed the following logs:

$ docker logs -f nginx
$ docker logs -f wordpress
$ docker exec wordpress tail -f /var/www/html/wp-content/debug.log

Logs were empty.

Next thing we tried was to run a bare-bones nginx server without the wordpress setup. So just run

$ docker run -d -p 80:80 nginx:alpine

Now the error vanished! This led us to believe that CloudFlare was working properly. So the problem had to be with wordpress. But then why was there no error in the logs? Without anything in the logs to give a clue, we were stuck for a long time. Then it dawned on us to access the VM directly using its private IP from the VPN and lo and behold the server responded and wordpress loaded up! So it couldn’t be a problem with wordpress!

When 2+2 does not equal 4: If CloudFlare is working properly as well as WordPress, then we are led to a logical contradiction and the 522 cannot be explained.

We then contacted CloudFlare when we were out of moves and they told us that all requests to the site were timing out leading them to believe there is something blocking requests from CloudFlare’s IP

Source IP: Y.Y.Y.Y
nc: connect to X.X.X.X port 443 (tcp) timed out: Operation now in progress
[exit code 1]
Source IP: Y.Y.Y.Y
nc: connect to X.X.X.X port 443 (tcp) timed out: Operation now in progress
[exit code 1]
Source IP: Y.Y.Y.Y
nc: connect to X.X.X.X port 443 (tcp) timed out: Operation now in progress
[exit code 1]
Source IP: Y.Y.Y.Y
nc: connect to X.X.X.X port 443 (tcp) timed out: Operation now in progress
[exit code 1]
Source IP: Y.Y.Y.Y
nc: connect to X.X.X.X port 443 (tcp) timed out: Operation now in progress
[exit code 1]

But we knew there was nothing blocking CloudFlare as the requests did succeed when we ran a bare-bones NGINX server.

Finally, the light struck us on the 4th day. The load balancer in Azure uses health probes to know if a machine is healthy. It construes any 200 response as indication of an unhealthy server. This is all documented here

An HTTP / HTTPS probe fails when:

Probe endpoint returns an HTTP response code other than 200 (for example, 403, 404, or 500). This will mark down the health probe immediately.

but it was the first time I was using a load balancer up close and personal and I didn’t know this. The endpoint it was using to test was / to which nginx was responding with a redirect 302. As soon as I changed the health probe to a custom endpoint to which I configured NGINX to return 200 the error vanished!

    # this special section is for the load balancer.
    # The elastic load balancer in azure needs to know if a machine is healthy. 
    # We assume it does that by making a request to the /health-probe endpoint.
    # If the load balancer gets a non-200 response it will mark the machine as unhealthy
    # and not send requests to it. 
    location /health-probe {
        return 200 OK;
    }

It was typical example when you need out-of-the-box thinking to fix a problem. For the longest time I kept thinking maybe the problem is in the docker network as I have struggled with it in the past. The fact that CouldFlare worked as well as WordPress (when we hit the VM using its private IP) but then why we were getting was the thing that puzzled me the most. It left us with no clue.

Posted in Software | Tagged | Leave a comment

Transactions and Locks in MySQL

Refer: https://stackoverflow.com/questions/61212438/why-do-we-need-to-lock-a-mysql-table-during-select-when-beginning-a-transaction/

TL;DR:

  1. MySQL docs are wrong
  2. Transactions act like a try-catch block. They don’t provide protection against concurrency. If something fails in a transaction, it can be rolled back to undo pending operations that were not committed.
  3. To handle concurrency you need to create locks.
  4. SELECT … LOCK IN SHARE MODE is a Reader-Writer lock that allows multiple threads to read and only 1 to write. https://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock
  5. SELECT … FOR UPDATE is a normal lock aka Mutex – mutual exclusion. https://en.wikipedia.org/wiki/Lock_(computer_science)

The statement

Beginning a transaction causes any pending transaction to be committed is extremely misleading because what it fails to mention is that The statements listed in this section (and any synonyms for them) implicitly end any transaction active in the current session, as if you had done a COMMIT before executing the statement. What this means is that if in our code you have a START TRANSACTION and you forget to explicitly COMMIT it, then later on whenever another START TRANSACTION is encountered, it will cause the running transaction to be committed before a new one can begin. This is likely not what you thought it does when you read  Beginning a transaction causes any pending transaction to be committed.
Readers–writer lock – Wikipedia
In computer science, a readers–writer (single-writer lock, a multi-reader lock, a push lock, or an MRSW lock) is a synchronization primitive that solves one of the readers–writers problems.An RW lock allows concurrent access for read-only operations, while write operations require exclusive access. This means that multiple threads can read the data in parallel but an exclusive lock is …
en.wikipedia.org
Posted in Software | Tagged | Leave a comment

A note on the dbDelta function

The dbDelta function documented at https://codex.wordpress.org/Creating_Tables_with_Plugins:

The dbDelta function examines the current table structure, compares it to the desired table structure, and either adds or modifies the table as necessary, so it can be very handy for updates (see wp-admin/upgrade-schema.php for more examples of how to use dbDelta)

except hat it does not work. Here is a real example when I tried to use it. I created a table using following command:

the table got created. But then I tried to update it using dbDelta

this is the result:

so $result = Array
(

[visitor_log(

] => Created table visitor_log(

)

whereas $wpdb->last_error = Table ‘visitor_log’ already exists

When I checked in mysql, the table is not modified in any way.

Posted in Software | Tagged | Leave a comment

Writing Your First WordPress Plugin

This post describes how to develop a WordPress plugin that can be used to count visits to a WordPress site. The basic idea is that we will create a table visitor_log in which we will store the timestamp, url_visited, ip_address, and user details of every visit. To do that, we need to:

  1. Create our plugin file (can name it anything ending with a .php) that will go under our plugin folder. The plugin folder itself is stored under wp-content/plugins. WordPress scans all folders under this directory and displays them in admin plugins dashboard. The plugin file should start with a preamble.

    Its also a good idea to declare a namespace to avoid conflicting function names. PHP uses the character in namespace instead of the more common and readable . character used in other languages. So com.foo.myclass becomes comfoomyclass in PHP

  2. Create the visitor_log table when plugin is activated
  3. Insert rows into the table when a page is visited

WP provides a register_activation_hook that can be used to do tasks when a plugin gets activated. We can use this hook as follows:

and then in our init_site_counter function, we can create a table like so

Finally for logging visits to the site, we subscribe to the template_redirect event

and add a row as shown below:

Exercise: The row_id variable is declared as 8 byte int which means the mysql table will be able to store

visits before the row_id variable overflows and resets to 1. In reality the disk is more likely to become full before the overflow happens. It would be good if we can auto-purge data from the table to prevent a disk-space full error. One elegant way to do this is to reduce the length of the row_id variable to lets say 4 bytes:

Now when the row_id overflows it will start overwriting the oldest records and prevent the table from growing in size which is exactly what we want except that we see an error when trying to insert a row with an id that already exists.

and if we remove the PRIMARY KEY from the table definition

there is an error when trying to create the table. It expects a primary key to be defined

Posted in Software | Tagged | 2 Comments