Category Archives: development

Web based or desktop based development, code snippets and code review goes in this area.

Cloning all github repositories of a user / organization

Github has become a new trend and people have started placing lots and lots of code on the website. So i have also been following this github trends and have pushed, cloned, forked multiple repositories. (total count of 72)

So today i wanted to create a backup of all github repositories under my account or under my organization account.I saw couple of solutions being provided here and here however most of the solutions are ruby based code.

So i thought lets see how much time and efforts it takes to write a simple python script to do the same. So here is the python version of github user/organization cloning script.

Script here

#!/usr/local/bin/python
import json
import requests
import argparse
import os
import sys
from git import Repo

def main(argv):
    global outp
    desc="""This program is used to  clone github repositories of a user / organization"""
    epilog="""Credit (C) Anant Shrivastava http://anantshri.info"""
    parser = argparse.ArgumentParser(description=desc,epilog=epilog)
    parser.add_argument("--name",help="User name",dest='target',required=True)
    parser.add_argument("--output",help="Output Directory",dest='out',required=False)
    x=parser.parse_args()
    target=x.target
    output=x.out
    if not output:
        output = os.path.curdir
    cnt=1
    while (cnt > 0):
        url="https://api.github.com/users/" + target + "/repos?page=" + str(cnt) + "&per_page=100"
        r=requests.get(url)
        js_data=json.loads(r.content)
        if len(js_data) == 0:
            print "No more repositories"
            cnt = -10
        else:
            print 'count: ' +  str(cnt) + ' : ' + str(len(js_data))
            if  "message" in js_data and "API rate limit exceeded" in js_data["message"]:
                print "Rate limit reached"
                cnt = -10
            else:
                for x in js_data:
                    git_url=x["clone_url"]
                    out_name=os.path.join(output, x["name"])

                    print git_url
                    Repo.clone_from(git_url, out_name)
        cnt=cnt+1


if __name__ == "__main__":
    main(sys.argv[1:])

Hope this can help someone.
Github url here

Example Usage
python -t github_clone_user.py --name anantshri --output "/tmp/test"

Also as it goes for anything publically available. It might contain errors and issues, don’t blame me if it blows your computer but if you do find ways to improve it or find better alternatives do drop a line in comments below.

P.S. at this point I needed user cloning stuff on priority so did that, organization cloning would be done but not today.

Plain Text Pages/Posts in WordPress

While experimenting with wordpress i was tasked with a situation where i am suppose to present some static text content on the website. However uploading a text file was out of question and the author wanted to keep everything controlled inside wordpress.

So i created a simple text only theme template for him and this combined with the plugin allowing custom permalinks provide us with options of having a custom name and still showing the wordpress page as static text only page.

Template Page is available online at : https://github.com/anantshri/wordpress_tricks/blob/master/plain_text_page.php

Source Code for viewing pleasure.

<?php
/*
* This templete will show all content as text only. in order to retain text only post while you save text in wordpress editor i would suggest add the text inside <pre></pre> tag
* the below code will remove any html / php tag that it will find in content and will only display the textual part of the file.
*/
header('Content-type: text/plain'); 
/**
 * Template Name: content text only
 * Description: Only Post content text only
 * @package WordPress 
 */
if (have_posts()) : 
	while (have_posts()) : the_post(); 
		$content=get_the_content();
		$cnt=strip_tags($content);
		echo $cnt;
	endwhile; // end of the loop. 
endif; 
?>

P.S. I suspect there might be better ways of dealing with this situation but my limited google skills didn’t provided a result in first 5 minutes so instead i spend 2 minutes on crafting this. If you do know a better way then feel free to suggest it via comments.

SVN Extractor for Web Pentesters

Many a times web application pen-testers are encountered with the presence of .svn folders. For those not aware .svn folder is used by SVN version control system to perform its operations. For a blackbox pentester this folder contains huge amount of information.

1) Uncover hidden files and folder names

2) Access the source code of the files.

3) download files even if the restrictions are in place at htaccess.

How this could be achieved.

1) Uncover hidden files and folder names

There are two ways in which this can be achieved based on the version of SVN in use.

for <1.6 we had .svn/entries files which contained list of files / folders as well as usernames used for commiting those files.

for >1.6 we have .svn/wc.db which contains simmilar data but in a sqlite3 format.

Those files could be directly accessible through url.

2) Access the source code / download files even if htaccess blocks its access.

SVN keeps a backup copy of all files in two seperate locations.

1) .svn/text-base/“filename”.svn-base

2) .svn/pristine/“XX”/“CHECKSUM”.svn-base

where

filename is actual name of file.

CHECKSUM is Sha1 sum of the file

XX is first two character of CHECKSUM.

first type of entries has one limitations suppose file name is testme.php so path becomes.

.svn/text-base/testme.php.svn-base

a large number of servers will execute the file using php engine and serve the output.

that’s where option 2 shines however this information is available only in case of wc.db (>1.6 SVN version) and this requires that .sv/pristine directory should be web accessible.

However after searching a lot i was not able to find a single code which can do both these things in one go.

so here is a tool which can perform both the operations in one script.

Usage

svn-extractor.py –url “url with .svn available”

Source Link : https://github.com/anantshri/svn-extractor

So far only tested on localhost environments however hoping to get some response on the same.

References

It would be unfair to say that i did all the research myself so here are the links to various resources i used to get the info out.

1) http://pen-testing.sans.org/blog/pen-testing/2012/12/06/all-your-svn-are-belong-to-us (manual technique for wc.db)

2) http://www.adamgotterer.com/post/28125474053/hacking-the-svn-directory-archive (manual technique for .svn/entries)

3) http://www.cirt.net/svnpristine (only automated tool i can find online doing 1/2 of what is in the tool)

MyLife : Hack : Yahoo Open HackDay 2011

Open Hack India

Open Hack India 2011

This was my first time attending Yahoo open hackday the event is all fun and a quick way to hacking onto the yahoo api’s.

I specifically focused on one API YQL, which basically claims to do the following


select * from internet

now that raises an eyebrow, in simpler term this is what is we can call a content scrapper’s dream, we get as less as 1000+ data tables which allow us to interact with various websites using well known sql standards.

here we can keep adding more and more tables if needed otherwise we can always revert back to generic tablees like

select * from rss where url =”http://blog.anantshri.info/feed/”

the thing that i like the most was they have given a direct access in the form of yql console you can check that by clicking the above link

http://developer.yahoo.com/yql/console/

I have been trying my luck to brew something for my self and a long lasting itch came back to me and i thought lets try solving the etch here.

so my hack for open hackday 2011 was : MY Life : a social content feed aggregation widget.

basically what i am doing is listed here in simplest terms.

1) take simple userid/username from users for various social networking sites.

2) create a unified feed based on user inputs

3) provide widget (HTML/JS) and PHP code to be used on site based on the user need.

So here is the hosted version of Hack

http://anantshri.info/openhack/mylife/

Note : YQL has rate limit and hence will only be able to fetch content for 10000 times a day. so if you do find output missing then its a good news for me basically my site has crossed 10000 users .

Database protection Techniques : a different prespective

Tips for Db Security

Disclaimer : This post keeps in mind the web frontends and web applications based attacks on DB Servers in mind.

  1. Any Userid used for web application connectivity should be clipped to specific ip addresses that could be localhost in case of same server usage for Db and App server. If two separate servers are used then clip the user id (s) with the application server ip address / hostname. Keep a strick log of who access the id and when.

  2. 1st entry in user /password data base should be a dummy entry with zero privilege and could be used to act as honeypot too.

  3. Default accounts to be removed / blocked.

  4. User Input validation should be a 3 step process.

    1. Web Page / Client Side validation : Jscript.

    2. Server (Application) : OWASP ESAPI or custom functions ocould be used.

    3. DB: use PL/SQL functions to strip input data.

  5. Another good utility to keep in mind is DBA_USERS_WITH_DEFPWD : contains list of users with default passwords, and with 11g all default accounts are locked.

  6. Web application developers should be provided with 3 different user level access to be used inside web application.

    1. Read Access : user with access to select query only.

    2. Write access: User with select update and delete access

    3. App_mod : access to write access plus drop and trunk.

Developers need to make sure the proper user access is used as and when required.

Note : I know a large number of developers might start crying on this that this will increase their headache but then in long run this could turn out to be a life saver.

Please share your opinion is this a good approach, is it going to help you. What do you thing should be added to this.

Tips for Db Security

  1. Any Userid used for web application connectivity should be clipped to specific ip addresses that could be localhost in case of same server usage for Db and App server. If two separate servers are used then clip the user id (s) with the application server ip address / hostname

  2. Web application developers should be provided with 3 different user level access to be used inside web application.

    1. Read Access : user with access to select query only.

    2. Write access: User with select update and delete access

    3. App_mod : access to write access plus drop and trunk.

Developers need to make sure the proper user access is used as and when required.

Note : I know a large number of developers might start crying on this that this will increase their headache but then in long run this could turn out to be a life saver.

  1. 1st entry in user /password data base should be a dummy entry with zero privilege and could be used to act as honeypot too.

  2. Default accounts to be removed / blocked.

  3. User Input validation should be a 3 step process.

    1. Web Page / Client Side validation : Jscript.

    2. Server (Application) : OWASP ESAPI or custom functions ocould be used.

    3. DB: use PL/SQL functions to strip input data.

  4. DBA_USERS_WITH_DEFPWD : contains list of users with default it.

    1. 11 g : all accounts will be locked and expiered byt default.

Howto add PPA in debian

I am back with some more scripting fun.

I have been working on configuring my new debian machine and found one utility very lacking in debian and that was add-apt-repository.
So i set down and took my time out and finally i am able to mix match this simple script.

Disclaimer: I know adding ppa can have adverse effects on debian machines

At this point the work that this script performs is

  1. add the repository of the ppa. (here i am using lucid as the distribution of choice, coz i am using squeeze as my debian version)
  2. add the gpg key to the keyring.

Disclaimer

  • This script is at this pointed tested only on one machine. debian Squeeze (mint Flavoured.)

File : add-apt-repository.sh

Steps to install this.

  1. Download file

$ wget http://blog.anantshri.info/content/uploads/2010/09/add-apt-repository.sh.txt

2.  Save this file in /usr/sbin/

$ cp add-apt-repository.sh.txt /usr/sbin/add-apt-repository

3.  Change permissions to execute

$ chmod o+x /usr/sbin/add-apt-repository

4.  Change ownership to root

$chown root:root /usr/sbin/add-apt-repository

5.  Now when ever you need to execute command type

$ sudo add-apt-repository ppa:ppa-name

Opening this script to larger audience so that we can crowdsource efforts if someone likes it.

hope this can help someone

File : add-apt-repository.sh

Change Log

7 – Jan – 2011 : Updated the tutorial to place the file @ /usr/sbin as suggested at various during comments.

6 – Aug – 2011 : Updated the script to deal with the security hole (although not easily exploitable) as suggested by 7eggert at comment no 23

10 Sep – 2011 : bin corrected to sbin in step 3 and 4..  : thanks to Craig for pointing that out

404-notifier modified for more details

 

Image by CyboRoZ

I have been using this great plugin by Alex King named as 404-notifier.

This plugin has one specific role and that is to find 404 error on your website and then notify it to you by RSS or E-mail.

But the e-mail generated are not of much use as it tells me only one part of story i.e. URL that was hit and got 404.

So i modified the code and now it makes a lot of sense and gives me some inside on what is actually happening.

So my latest code and send out messages with following details. (providing Pic of the sample message.)

and i have been using this for past 1 yr on my blog. so what exactly i can get from this extra info.

If the link is not refereed by any page then it means that link is either marked as bookmark or some automated bot is trying to access the URL.

User Agents tells a lot more about who is accessing the link.

and just in case you don’t what to be irritated by a perticular user you always have his I.P. address to block him in htaccess.

So i have launched the modifed code at 404-notifier @ google code

hope this code can help someone.





Tweets in wordpress Posts

While Surfing the web i came across a forum thread @ net builders asking for some wordpress optimization option.

So i surfed the wordpress Plugin Repository and then found this simple plugin Twitter for wordpress created by Ricardo González

So i went up and customized it as per the need as requested in forum and here we are : tweet-in-wordpress tweet-in-wordpress-0.2

A simple Tweets in post plugin with a brand new shortcode.

Continue reading