Author Archives: bikrant

Linux tuning : Network performance

Here are few setting that can greatly improve network performance of your Linux machine:

# set OS max read & write buffers
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

#TCP Autotuning settings
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216

# disable slow start after idle
net.ipv4.tcp_slow_start_after_idle = 0

#enable TCP window scaling
net.ipv4.tcp_window_scaling = 1

With these settings in place, we stopped seeing all the Netty reconnect problems in the Storm topologies. Make sure to apply these settings to all Storm Supervisor nodes.

These settings can be applied at the run time using sysctl command as, but this will not persist between the system reboots:

$ sudo sysctl -w net.ipv4.tcp_rmem = "4096 87380 16777216"

To permanently save them update /etc/sysctl.conf or a file under /etc/sysctl.d, and run this command:
$ sudo sysctl -p /etc/sysctl.conf

 

Get current Epoch timestamp

#java:

System.currentTimeMIllis();
Note: the value is in milliseconds. Make sure to divide by 1000 if you need the value in seconds

#php:

time()

#objective-c:

NSDate now = [NSDate date];
NSTimeInterval nowEpochSeconds = [now timeIntervalSince1970];

# perl:

time

# python:

import

int(time.time())

#.NET C#

# MySQL:

SELECT unix_timestamp(now());

# Unix/Linux shell:

$ date +%s

#

Multiprocessing jobs with worker pool in Python

#multiprocessing jobs with worker pool in python

from multiprocessing import Process, Lock, Pool
import time
import datetime
import sys
import os
from subprocess import Popen, PIPE

def proc(metric):
    print "Working on %s"%(metric)
    cmd1 = "/usr/bin/tsdb scan --import 2012/04/10 sum %s"%(metric)
    cmd2 = "grep %s"%(metric)

    logfile = open("%s.dat.gz"%(metric), 'w')
    p1 = Popen(cmd1.split(" "), stdout=PIPE)
    p2 = Popen(cmd2.split(" "), stdin=p1.stdout, stdout=PIPE)
    p3 = Popen(['gzip'], stdin=p2.stdout, stdout=logfile)
    p3.communicate()
    logfile.close()

if __name__ == '__main__':
    f = open('file_list_of_metrics.txt', 'r')
    l = [i.rstrip() for i in f]
    pool = Pool(processes=10)
    pool.map(proc, l, 1)
    pool.close()
    pool.join()

Create new maven project from scratch

1. Create a new project
$ mvn archetype:generate -DgroupId=com.xyz.myapp -DartifactId=MyApp -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
$ cd MyApp

2. To Build Package:
$ mvn package
//this will create target/MyApp-1.0-SNAPSHOT.jar

3. To Execute
$ java -cp target/MyApp-1.0-SNAPSHOT.jar com.xyz.myapp

# Adding dependency to your project
Let say you wanted to add “json-simple” to your project:
add inside in the pom.xml file:

<dependency>
 <groupId>com.googlecode.json-simple</groupId>
 <artifactId>json-simple</artifactId>
 <version>1.1</version>
</dependency>

– Now to execute a package with dependency you will have to include the path to the jar (I think there is a way to do it with pom.xml, but I don’t know this yet)
– This could complicated if you have multiple dependencies
– And also need to make sure you have the pkgs in the system where you want to deploy your code
– Solution: Create fat jar (a jar that includes all dependent jars, side effect: output file is bigger and takes longer to run ‘mvn package’ command)
– To create fat jar add this to your pom.xml inside :

<build>
 <plugins>
  <plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-assembly-plugin</artifactId>
   <version>2.2.1</version>
   <configuration>
    <descriptorRefs>
     <descriptorRef>jar-with-dependencies</descriptorRef>
    </descriptorRefs>
   </configuration>
   <executions>
    <execution>
     <id>assemble-all</id>
     <phase>package</phase>
     <goals>
      <goal>single</goal>
     </goals>
    </execution>
   </executions>
  </plugin>
 </plugins>
</build>