Shell script to change files content on multiple git repository

Continuing last post, change content in multiple files, I made another script to change multiple files on each git repository and push the code.

1. Make folder list in array

#!/bin/sh

array=( site1 site2 site3 )

2. For each loop for each sites

for site in "${array[@]}"
do
  #some command
done

3. Replace content in each file

cd ~/home/"$site"/
rpl ".title" ".h1_title" ~/home/"$site"/css/common.css

4. Commit the changes

git pull
git add .
git commit -am 'title changed to h1 title on css file'
git push

5. This is the final code.

#!/bin/sh

array=( site1 site2 site3 )

for site in "${array[@]}"
do
  cd ~/home/"$site"/
  rpl ".title" ".h1_title" ~/home/"$site"/css/common.css
  git pull
  git add .
  git commit -am 'title changed to h1 title on css file'
  git push
done

This is especially useful when I have many sites with same structure.

 

Change content text on multiple files using shell script

When I want to change text in several files, it’s hard to open each single file and replace.
So I made one shell script to do the job.
Here’s one example using sed command.

find /home/user/directory -name \*.* -exec sed -i “s/search_text/replace_text/g” {} \;

BTW, it’s hard to remember. Also if I want to change including special character, it’s more harder. like this one. <?php echo $text;?> 

I searched for easy way and found good command. rpl – replace string in files

It’s really easy to use. If you don’t have rpl installed, install it using brew.

$ brew install rpl
rpl "<?php echo $search_text;?>" "<?php echo $replace_text;?>" /home/user/directory

Amazing! I don’t need to remember complicate command line.
However, I want to change in multiple files at the same time, moreover I would like to make easier input without change core programming.

#!/bin/sh
echo "Please enter search: "
read search_variable
echo "Search: $search_variable"
echo "Please enter replace: "
read replace_variable
echo "Replace: $replace_variable"

This bash script will get user input, both find and replace text.

for file in `find . -type f`
do
    rpl "$search_variable" "$replace_variable" $file
done

And this will find all files and replace text. Here’s final code. Save to replace.sh file in good place.

#!/bin/sh
echo "Please enter search: "
read search_variable
echo "Search: $search_variable"
echo "Please enter replace: "
read replace_variable
echo "Replace: $replace_variable"


for file in `find . -type f`
do
    rpl "$search_variable" "$replace_variable" $file
done

One more, I would like to use this command from every where.

alias replace="~/home/shell/replace.sh"

Put this in the .bash_profile, so the comman is available from any where.

 

Real time user count from Google analytics API

I posted how to say hello to Google Analytics API.

Continuing the post, I would like to get Real-time user number using Google Analytics API.

1. Initialize google analytics API.

require '../composer/vendor/autoload.php';

function initializeAnalytics(){
  // Creates and returns the Analytics Reporting service object.

  // Use the developers console and download your service account
  // credentials in JSON format. Place them in this directory or
  // change the key file location if necessary.
  $KEY_FILE_LOCATION = __DIR__ . '/credentials.json';

  // Create and configure a new client object.
  $client = new Google_Client();
  $client->setApplicationName("Hello Analytics Reporting");
  $client->setAuthConfig($KEY_FILE_LOCATION);
  $client->setScopes(['https://www.googleapis.com/auth/analytics.readonly']);
  $analytics = new Google_Service_Analytics($client);

  return $analytics;
}

$analytics = initializeAnalytics();

2. Function for Real-time user number.

function get_realtime_active_user($analytics, $ga_internal_id){
  $optParams = array(
      'dimensions' => 'rt:medium');
  try {
    $results = $analytics->data_realtime->get(
        'ga:'.$ga_internal_id,
        'rt:activeUsers',
        $optParams);
    // Success. 
    $return = $results->totalsForAllResults['rt:activeUsers'];
    return $return;
  } catch (apiServiceException $e) {
    // Handle API service exceptions.
    $error = $e->getMessage();
  }
  
}

3. Get multiple site’s data, change $ga_id_array variable according to your GA account number.
You can get GA account number just like the below.

ga account number
ga account number copy from here
$data = array();
$ga_id_array = array('site1'=>'1234567','site2'=>'12345678','site3'=>'123456789');
foreach($ga_id_array as $name => $ga_id){
  $each_data = array();
  $each_data['name'] = $name;
  $each_data['num'] = get_realtime_active_user($analytics, $ga_id);
  $data[] = $each_data;
}
echo json_encode($data);

4. All together.

<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
  
require '../composer/vendor/autoload.php';

function initializeAnalytics(){
  // Creates and returns the Analytics Reporting service object.

  // Use the developers console and download your service account
  // credentials in JSON format. Place them in this directory or
  // change the key file location if necessary.
  $KEY_FILE_LOCATION = __DIR__ . '/credentials.json';

  // Create and configure a new client object.
  $client = new Google_Client();
  $client->setApplicationName("Hello Analytics Reporting");
  $client->setAuthConfig($KEY_FILE_LOCATION);
  $client->setScopes(['https://www.googleapis.com/auth/analytics.readonly']);
  $analytics = new Google_Service_Analytics($client);

  return $analytics;
}

$analytics = initializeAnalytics();

function get_realtime_active_user($analytics, $ga_internal_id){
  $optParams = array(
      'dimensions' => 'rt:medium');
  try {
    $results = $analytics->data_realtime->get(
        'ga:'.$ga_internal_id,
        'rt:activeUsers',
        $optParams);
    // Success. 
    $return = $results->totalsForAllResults['rt:activeUsers'];
    return $return;
  } catch (apiServiceException $e) {
    // Handle API service exceptions.
    $error = $e->getMessage();
  }
  
}

$data = array();
$ga_id_array = array('site1'=>'1234567','site2'=>'12345678','site3'=>'123456789');
foreach($ga_id_array as $name => $ga_id){
  $each_data = array();
  $each_data['name'] = $name;
  $each_data['num'] = get_realtime_active_user($analytics, $ga_id);
  $data[] = $each_data;
}
echo json_encode($data);

That’s all, this will return with JSON format like this.

[{"name":"site1","num":"722"},{"name":"site2","num":"100"},{"name":"site3","num":"20"}]

 

Get started with Google analytics API

Making something new is always challenging.
Google API is well made, but documentation is always confusing me.
So let’s say “Hello” to Google Analytics. It’s good start for this kind of complicate API work.

I have checked this Hello analytics page.

1. Download composer

$ curl -sS https://getcomposer.org/installer | php

2. creat composer.json

{
  "require": {
    "google/apiclient": "^2.0"
  }
}

3. Initialize composer with google analytics. This will create vendor folder with google folder inside

$ php composer.phar install

4. Get credential JSON file from Google credential page.
Go to Credentials > Create credentials > Service account key > Click key type as JSON
Download and save it in to proper folder.

5. Enable Google Analytics API
Go to Library on Google API page. And click Analytics API under Other popular APIs

6. Open Json credential file and copy client email and Add user profile in google analytics ( admin > user management > Add permissions for:)
e.g. some-username@my-api-123456.iam.gserviceaccount.com

7. Write this code and run it from browser.

<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
require '../composer/vendor/autoload.php';

function initializeAnalytics(){
  // Creates and returns the Analytics Reporting service object.
  // Use the developers console and download your service account
  // credentials in JSON format. Place them in this directory or
  // change the key file location if necessary.
  $KEY_FILE_LOCATION = __DIR__ . '/credentials.json';

  // Create and configure a new client object.
  $client = new Google_Client();
  $client->setApplicationName("Hello Analytics Reporting");
  $client->setAuthConfig($KEY_FILE_LOCATION);
  $client->setScopes(['https://www.googleapis.com/auth/analytics.readonly']);
  $analytics = new Google_Service_Analytics($client);

  return $analytics;
}

function getFirstProfileId($analytics) {
  // Get the user's first view (profile) ID.

  // Get the list of accounts for the authorized user.
  $accounts = $analytics->management_accounts->listManagementAccounts();

  if (count($accounts->getItems()) > 0) {
    $items = $accounts->getItems();
    $firstAccountId = $items[0]->getId();

    // Get the list of properties for the authorized user.
    $properties = $analytics->management_webproperties
        ->listManagementWebproperties($firstAccountId);

    if (count($properties->getItems()) > 0) {
      $items = $properties->getItems();
      $firstPropertyId = $items[0]->getId();

      // Get the list of views (profiles) for the authorized user.
      $profiles = $analytics->management_profiles
          ->listManagementProfiles($firstAccountId, $firstPropertyId);

      if (count($profiles->getItems()) > 0) {
        $items = $profiles->getItems();

        // Return the first view (profile) ID.
        return $items[0]->getId();

      } else {
        throw new Exception('No views (profiles) found for this user.');
      }
    } else {
      throw new Exception('No properties found for this user.');
    }
  } else {
    throw new Exception('No accounts found for this user.');
  }
}

function getResults($analytics, $profileId) {
  // Calls the Core Reporting API and queries for the number of sessions
  // for the last seven days.
   return $analytics->data_ga->get(
       'ga:' . $profileId,
       '7daysAgo',
       'today',
       'ga:sessions');
}

function printResults($results) {
  // Parses the response from the Core Reporting API and prints
  // the profile name and total sessions.
  if (count($results->getRows()) > 0) {

    // Get the profile name.
    $profileName = $results->getProfileInfo()->getProfileName();

    // Get the entry for the first entry in the first row.
    $rows = $results->getRows();
    $sessions = $rows[0][0];

    // Print the results.
    print "First view (profile) found: $profileName\n";
    print "Total sessions: $sessions\n";
  } else {
    print "No results found.\n";
  }
}

$analytics = initializeAnalytics();
$profile = getFirstProfileId($analytics);
$results = getResults($analytics, $profile);
printResults($results);

This looks easy but it need some effort to make it work.
Once you get Hello from analytics, next step is a lot easier.

 

Optimize CSS asynchronous load

One of way to make website loading faster is CSS async load.
It need to use Javascript to load CSS after loading.

There’s a few steps to make it work.

1. Make css file to Json data format.
Let’s take this css file for example.
http://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/icons.css
It contains css code like this.

@charset "UTF-8";

/* untitled-font-1 */
@font-face {
  font-family: "untitled-font-1";
  src:url("https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.eot");
  src:url("https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.eot?#iefix") format("embedded-opentype"),
    url("https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.woff") format("woff"),
    url("https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.ttf") format("truetype"),
    url("https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.svg#1491933871") format("svg");
  font-weight: normal;
  font-style: normal;
}

[data-icon]:before {
  font-family: "untitled-font-1" !important;
  content: attr(data-icon);
  font-style: normal !important;
  font-weight: normal !important;
  font-variant: normal !important;
  text-transform: none !important;
  speak: none;
  line-height: 1;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}
.
.
.

 

Let’s minify it from cssminifier.com

@charset "UTF-8";[class*=" icon-"]:before,[class^=icon-]:before,[data-icon]:before{font-family:untitled-font-1!important;font-style:normal!important;font-weight:400!important;font-variant:normal!important;text-transform:none!important;speak:none;line-height:1;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}@font-face{font-family:untitled-font-1;src:url(https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.eot);src:url(https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.eot?#iefix) format("embedded-opentype"),url(https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.woff) format("woff"),url(https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.ttf) format("truetype"),url(https://file.myfontastic.com/eNzJBJA24QdpMtRZDEqjmX/fonts/1491933871.svg#1491933871) format("svg");font-weight:400;font-style:normal}[data-icon]:before{content:attr(data-icon)}.icon-chevron-right:before{content:"\61"}.icon-chevron-left:before{content:"\62"}.icon-twitter-alt:before{content:"\64"}.icon-youtube:before{content:"\63"}.icon-facebook:before{content:"\65"}.icon-pinterest-circled:before{content:"\66"}.icon-search:before{content:"\69"}.icon-star-1:before{content:"\6b"}.icon-instagram:before{content:"\6c"}.icon-rss:before{content:"\68"}.icon-googleplus:before{content:"\67"}.icon-pinterest:before{content:"\6a"}.icon-remove:before{content:"\6d"}.icon-arrow-circle-right:before{content:"\6e"}.icon-bars:before{content:"\6f"}.icon-share-alt:before{content:"\70"}

It became much smaller.
Using editor, escape backslash (\) like this (\\). And then escape double quote (“) to (\”). Also escape slash (/) to (\/)

Then it will look like this.

@charset \"UTF-8\";[class*=\" icon-\"]:before,[class^=icon-]:before,[data-icon]:before{font-family:untitled-font-1!important;font-style:normal!important;font-weight:400!important;font-variant:normal!important;text-transform:none!important;speak:none;line-height:1;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}@font-face{font-family:untitled-font-1;src:url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.eot);src:url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.eot?#iefix) format(\"embedded-opentype\"),url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.woff) format(\"woff\"),url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.ttf) format(\"truetype\"),url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.svg#1491933871) format(\"svg\");font-weight:400;font-style:normal}[data-icon]:before{content:attr(data-icon)}.icon-chevron-right:before{content:\"\\61\"}.icon-chevron-left:before{content:\"\\62\"}.icon-twitter-alt:before{content:\"\\64\"}.icon-youtube:before{content:\"\\63\"}.icon-facebook:before{content:\"\\65\"}.icon-pinterest-circled:before{content:\"\\66\"}.icon-search:before{content:\"\\69\"}.icon-star-1:before{content:\"\\6b\"}.icon-instagram:before{content:\"\\6c\"}.icon-rss:before{content:\"\\68\"}.icon-googleplus:before{content:\"\\67\"}.icon-pinterest:before{content:\"\\6a\"}.icon-remove:before{content:\"\\6d\"}.icon-arrow-circle-right:before{content:\"\\6e\"}.icon-bars:before{content:\"\\6f\"}.icon-share-alt:before{content:\"\\70\"}

Let’s wrap this with this javascript function. callbackCSS({"data":"json content goes here"})

Final myfontastic.json file look like this.

callbackCSS({"data":"@charset \"UTF-8\";[class*=\" icon-\"]:before,[class^=icon-]:before,[data-icon]:before{font-family:untitled-font-1!important;font-style:normal!important;font-weight:400!important;font-variant:normal!important;text-transform:none!important;speak:none;line-height:1;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}@font-face{font-family:untitled-font-1;src:url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.eot);src:url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.eot?#iefix) format(\"embedded-opentype\"),url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.woff) format(\"woff\"),url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.ttf) format(\"truetype\"),url(https:\/\/file.myfontastic.com\/eNzJBJA24QdpMtRZDEqjmX\/fonts\/1491933871.svg#1491933871) format(\"svg\");font-weight:400;font-style:normal}[data-icon]:before{content:attr(data-icon)}.icon-chevron-right:before{content:\"\\61\"}.icon-chevron-left:before{content:\"\\62\"}.icon-twitter-alt:before{content:\"\\64\"}.icon-youtube:before{content:\"\\63\"}.icon-facebook:before{content:\"\\65\"}.icon-pinterest-circled:before{content:\"\\66\"}.icon-search:before{content:\"\\69\"}.icon-star-1:before{content:\"\\6b\"}.icon-instagram:before{content:\"\\6c\"}.icon-rss:before{content:\"\\68\"}.icon-googleplus:before{content:\"\\67\"}.icon-pinterest:before{content:\"\\6a\"}.icon-remove:before{content:\"\\6d\"}.icon-arrow-circle-right:before{content:\"\\6e\"}.icon-bars:before{content:\"\\6f\"}.icon-share-alt:before{content:\"\\70\"}"})

And define this javascript function on the top of your page. You can include this in common javascript file on the header.

var LSYE=function(h){var s=document.createElement('script');s.src=h;document.getElementsByTagName('head')[0].appendChild(s);};
var callbackCSS=function(d){var o=document.createElement("style");o.innerHTML=d.data;document.getElementsByTagName('head')[0].appendChild(o);};

LYSE function is put javascript to header async way.
And callbackCSS will put css data to header after that.

So you can put this line at the bottom of page, this css will be loading asynchronous way.

<script>LSYE("myfontastic.json");</script>

You can add async attribute to loading file during page load.

<script async>LSYE("myfontastic.json");</script>

Or add defer attribute to loading file after page load finished.

<script async>LSYE("myfontastic.json");</script>

In my case, this css file contains icons for the site. So icons are appeared just few seconds later, but shows totally fine.
This will be helpful when the file is not loading or slow. So this doesn’t affect to site speed.

 

Amazon S3 folder open to public with bucket policy

When upload file to S3, files are not allowed to open to public.
We need to make policy to open public.

There’s Bucket Policy on permission tab.
Add policy with Json format like following:

{
    "Version": "2012-10-17",
    "Id": "AWS-some-name-of-id",
    "Statement": [
        {
            "Sid": "AWS-some-name-of-id",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::arn-url-get-from-above/*"
        }
    ]
}

We need to keep Version date, it’s not today’s date, but it is Amazon policy version date.
Also Resource ARN code could get from above the policy editor.

Varnish command

Varnish is web application to increase web contents delivery time by cache. It can be installed in front of any servers with HTTP. Varnish is very fast, so it can speed up 300 to 1000 times faster.

Using Ubuntu, here’s varnish config file located:

vi /etc/varnish/default.vcl

After config varnish vcl file, you can test it using this command:

varnishd -C -f /etc/varnish/default.vcl

To varnish start / stop / restart:

service varnish start
service varnish stop 
service varnish restart

To view varnish status:

service varnish status

In my case, varnish stopped working with no reason.
I found out that varnish disk was full, so I had to find out which file has the most big size.

du -h / | grep '[0-9\.]\+G'

And I narrow down and found varnish log file was too big.
After delete the file and start, it worked as normally.

Happy caching!
Bravo Varnish!

Crontab command – working status, log, cron job

When setup crontab on the server, it’s hard to check whether it’s working properly or not.

There’s command to check the status and log.

To check cron status and see the latest cronjob:

/etc/init.d/cron status

To start running:

service cron start

To stop running:

service cron stop

To restart running:

service cron restart

Cronjob example:

Running php script every minute:

* * * * * /usr/bin/php -e /websites/example.php > /dev/null 2>&1

/dev/null  this line is added for no log

If you want to see running log:

* * * * * /usr/bin/php -e /websites/example.php > /home/user/cron.log 2>&1

/dev/null  this line is added for saving to log file

Running php script every 2 minutes, which will run 30 times per hour:

*/2 * * * * /usr/bin/php -e /websites/example.php > /dev/null 2>&1

Running php script every 10 minutes, which will run 6 times per hour:

*/10 * * * * /usr/bin/php -e /websites/example.php > /dev/null 2>&1

Running php script designated minutes, this will running every 1,11,21,31,41,51 minutes:

1,11,21,31,41,51 * * * * /usr/bin/php -e /websites/example.php > /dev/null 2>&1

Running php script once a day, this will running at 1:00am every day:

0 1 * * * /usr/bin/php -e /websites/example.php > /dev/null 2>&1

Let’s keep the time well.

Mysql error 1364 Field doesn’t have a default values

When get this error: mysql error 1364 Field doesn’t have a default values.

This happens when there’s key index but no values given in STRICT_TRANS_TABLES mode.

So we need to change sql_mode which is defined in my.cnf file.

When I search about the file, unfortunately I couldn’t find the file.

So rather than modify file, use query to change mysql configuration:

SET GLOBAL sql_mode = 'ONLY_FULL_GROUP_BY,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'

If this doesn’t work or it doesn’t allow you to query, you can modify mysql.cnf file.

vi /etc/mysql/conf.d/mysql.cnf

Then add these line

[mysqld] sql_mode = "ONLY_FULL_GROUP_BY,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

And reboot mysql.

sudo service mysql restart

If still not working, add this disable_strict_mode.cnf file below location.

vi  /etc/mysql/conf.d/disable_strict_mode.cnf

Then add these 2 lines.

sql_mode = ONLY_FULL_GROUP_BY,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

After reboot, check it by running this query:

SHOW VARIABLES LIKE 'sql_mode';