CategoryUncategorized

Web Spiders and expired content

All my spiders were taking all content from a website on a single visit starting from the begining.

It seems that the idea of remembering which urls “produce” links with content is not so very bad.

Here is what I found for diri.bg – a local Bulgarian SE.

I see that diri.bg hasn’t remove from their page

show_categories.php

Even I have no links to this page. Check the result here

Ops. Google do it the same way: here

Then how to get rid of old pages without leaving “bad” links in internet?

I will try to put show_categories.php in robots.txt to see what will happen with this page.

Search engines

It seems that I know too little for the world of the search engines. I will try to summarize the search engines that can be used for real applications. Continue reading

Thinks that I want to read

http://scpd.stanford.edu/knuth/

Rails plugins, gems, search engines, applications

Plugins & gems lists

Associations

http://www.workingwithrails.com/railsplugin/4801-has-many-polymorphs

Versioning of AR

http://github.com/fatjam/acts_as_revisable/tree/master

http://opensoul.org/2006/7/21/acts_as_audited

Scafolding

http://streamlinedframework.org/pages/about

Ajax pagination with JQuery

http://ozmm.org/posts/ajax_will_paginate_jq_style.html

Find_by_param is a nice and easy way to handle permalinks and dealing with searching for to_param values

http://github.com/bumi/find_by_param/tree/master

Asset Plugin – better than rails 2.0 integrated

Asset Packager

Image Magic

http://vantulder.net/rails/magick/

OpenID

http://wiki.rubyonrails.org/rails/pages/OpenidLoginGenerator

http://github.com/technoweenie/restful-authentication/tree/master

http://github.com/mrflip/

Model graph visualize

Article on franzens.org

  • http://visualizemodels.rubyforge.org/
  • http://rav.rubyforge.org/
  • http://railroad.rubyforge.org/

Simple Captcha

http://expressica.com/simple_captcha/

Memcached stuff

http://townx.org/rails_and_memcached

Good rails 2.1 overview

memcached -vv -l 127.0.0.1 -p 11211 -m 256 -P /tmp/memcached.pid -u mongrel

memcached-tool 127.0.0.1

monitor the connections with the hidden option in the memcached-tool

echo $(($(netstat -nt | grep 11211 | grep -v WAIT | wc -l)/2)); ./memcached-tool 127.0.0.1:11211 dump > memdump; cat memdump

memcached-tool

alternative memcached client

PDF Output

http://ruby-pdf.rubyforge.org/pdf-writer/

http://rubyforge.org/projects/railspdfplugin/

Somone example report

Need to be checked: http://code.rubyreports.org/

Reporting

ruport & acts_as_reportable
some java reporting into rails

Sending Email

Inline email attachments plugin: http://flow.handle.it/past/2007/11/5/inline_attachment_now_official_rocks/

Wiki integration

http://github.com/queso/signal-wiki/tree/master

Repositories

http://github.com/mrflip/

Tutorials

Rails2.0 Video http://www.vimeo.com/425800

http://media.rubyonrails.org/video/rails_take2_with_sound.mov

http://akitaonrails.com/2007/12/12/rolling-with-rails-2-0-the-first-full-tutorial

http://akitaonrails.com/2007/12/12/rolling-with-rails-2-0-the-first-full-tutorial-part-2

Performance and Loging

Announcing Clientperf: Simple Client-Side Rails Performance

web statistics

http://www.railstat.com/wiki/FAQ

http://haveamint.com/about/feature_highlights – not free

production log analyzers

webstat like: http://watson.rubyforge.org/

speed: http://github.com/wvanbergen/request-log-analyzer/wikis

speed: http://ckhsponge.wordpress.com/2006/10/11/ruby-on-rails-log-analyzer-rawk/

pl-analyse: http://seattlerb.rubyforge.org/production_log_analyzer http://seattlerb.rubyforge.org/production_log_analyzer

Open source rails projects

Server setup

capistrano

mod_rails

  • http://www.sysadminschronicles.com/articles/2008/05/13/ubuntu-8-04-rails-server-using-passenger-part-2

nginx

  • check my post
  • gem install nginx_config_generator

Everyday SQL statements

Tools

Status

SHOW status where Variable_name like ‘Th%’ or Variable_name like ‘%Connec%’ ;
SHOW [GLOBAL | SESSION] STATUS [LIKE ‘pattern’ | WHERE expr]

Check/Repair tables

mysqlcheck -u root -p***** –auto-repair –check –optimize –all-databases

Profiling

watch -n 0.5 ‘mysql -u root -ppass -e “SHOW FULL PROCESSLIST” | grep Query’

http://opendba.blogspot.com/2008/03/mysql-finally-ability-to-traceprofile.html

mysql> set profiling=1;
mysql> select count(*) from mysql.user;
mysql> show profile;

Dump

pg_dump -U test arachnid_archiv_test –inserts -h chaos.spider.bg –encoding=utf8 -f pgsql.sql

mysqldump -c -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS -r $tfile --add-drop-table $DB
mysqldump -c -h localhost -u system3 system3_production -psomepassword -r system3_production.sql  --add-drop-table $DB

Dump for full backup with flushing of the log files

mysqldump -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS
–single-transaction –all-databases –delete-master-logs –flush-logs –master-data=2
> backup_sunday_1_PM.sql

Encoding problems

http://www.hostbulgaria.com/tutorials/mysql-charset-encoding.aspx

SHOW VARIABLES LIKE ‘character_set_%’;
curl -i http://system3.spider.bg

Creating a database

create database re_production DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

Creating a user

GRANT ALL PRIVILEGES ON arachnid_production.* TO ‘payak’@’%’ IDENTIFIED BY ‘payakpassword’ WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON system3_production.* TO ‘payak’@’%’ IDENTIFIED BY ‘payakpassword’ WITH GRANT OPTION;

mysqladmin -u [user] -h localhost -p password ‘[new_password]’

SQL for a table

SHOW CREATE TABLE tblname;

mysql tunel to another machine

ssh -N -f -l root -L 0.0.0.0:3307:91.196.240.132:3306 s1
open port 3307 on the local machine to 91.196.240.132:3306 and login into s1 with root

Replace text

UPDATE script_histories SET cod_script = replace(cod_script,”observer.ArchiveObserver(siteId)”,”observer.ArchiveObserver(siteId, script_id, owned_source_id)”);

Copy from one table to another

DELETE FROM system3_production.articles;
INSERT INTO system3_production.articles SELECT * FROM arachnid_from_screen.articles;

Sessions for Rails

select count(*) from sessions where updated_at < DATE_SUB(now(), INTERVAL 3 DAY);

Binnary loging

http://dev.mysql.com/doc/refman/5.0/en/recovery-from-backups.html

Check this attachment here: mysql-presentation on replication etc.

  • See the status of the log files
    SHOW BINARY LOGS;
    SHOW MASTER STATUS;
  • Clean the binary logs instantly
    RESET MASTER;
  • Clean binary logs to date/name
    PURGE BINARY LOGS TO ”mysqld-bin.00XXXX’;
  • Configurations in my.cnf
    log-bin
    server-id = 1
    expire_logs_days = 1
    max_binlog_size = 100M

Configuration

max_allowed_packet = 50M
wait_timeout=720
max_connections=1000
connect_timeout=20

query_cache_limit=8M #~~~ removed, 1M def. max pozwl. razmer za cache-hirane na edna zajawka
query_cache_size=128M #~~~ 32M, 0 def.
query_cache_type=1

Restoring the maintian Debian User

GRANT ALL PRIVILEGES ON *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY '<password>' WITH GRANT OPTION;
Replace <password> with your debian-sys-maint password.

Game

Once in the dark DOS ages I have one game with which you can make wonderfull animations. In the “Windows” era I miss this game a lot….until I found this nice replacement.

http://phun.cs.umu.se/wiki/Download

Масапан

Незнам защо но успях да открия в интернет само тази рецепта за тези вкусни бонбони и се изуших да я запазя за ентусястите, в момента бадемите са в хладилника и побеляват активно.

200 гр. сурови бадеми
90 гр. захар
130 гр. вода
1 ч. л. лимонов сок

Бадемите се накисват във вряла вода, която да ги покрие два пръста. Престояват така около 20-30 минути и се обелват, като се натискат леко в края на ядката.
Слагат се върху домакинска хартия и така престояват една нощ в хладилника, за да се изтегли водата от тях и за да побелеят.
На другият ден бадемите се смилат фино в машинка за кафе или в кухненски комбайн, до консистенция почти на брашно. За по-добър резултат се слагат по малко бадеми, защото при по-големи количества долния пласт става на паста, а горният е едро смлян. Прави се захарен сироп от захарта и водата по следния начин-слагат се двете съставки в дълбок съд и се оставя да заври сместа. После се намаля котлона на средна степен и се оставя сиропа да ври докато не стане със следната консистенция-като се капне капка от него в чаша със студена вода, то капката трябва да падне цяла на дъното,
Когато този момент (Punto di masapan) е достигнат, смлените бадеми се прибавят и вече на съвсем слаб котлон се превъртат заедно със сиропа, докато не стане хомогенна тестообразна смес. Тенджерата се отсранява от котлона, хубаво се смесва с лимовият сок и се приготвят бонбони с помощта на метална формичка и найлоново опаковъчно домакинско фолио-фолиото се слага във формичката, отгоре се натъпква плътно сместта и се обръщат така получените бонбони върху пекарска домакинска хартия.
След като се стегнат се нареждат в празна картонена кутия от бонбони и преседяват още една нощ в хладилника.
На другият ден можете да опитате този невероятно вкусен еврейски сладкиш, който се приготвя за сватби, раждане на деца, Брит мила или Бар или Бат мицва.
Той има особен статут в еврейската култура, защото приготвянето му отнема много време и е относително скъп за направа.

Източник
masapan

Software and Tutorial Links

IPTables
Linux firewall with packet string-matching support
http://www.securityfocus.com/infocus/1531
Firewall scripts
http://linuxfirewall.start.bg/
http://www.puschitz.com/FirewallAndRouters.shtml
Debian Kernel how-to
http://myfreesoft.net/phpBB2/viewtopic.php?t=52 * in Bulgarian *
Source: http://www.howtoforge.com/forums/showthread.php?t=21
http://www.howtoforge.com/kernel_compilation_debian
Active Ports (Windows)
http://www.microsoft.com/technet/sysinternals/Networking/TcpView.mspx
Security and Audits Checks & updates
http://sourceforge.net/projects/advchk
RedHat – http://www.redhat.com/spacewalk/

Avoid common mistake in web programming

WebGoat

WebGoat is a deliberately insecure J2EE web application designed to teach web application security concepts

WebGoat is java web application which can show you a lot of the web attacks that can be performed on your website. It is a must have for all web developers.

You can download it from sourceforge or from google

The package contains a tomcat server and you can run it from the distribution. But in my case there was only windows configuration files so I have to take the war file and put in the standalone tomcat installation. Also move the permissions for the roles/users to my tomcat installation.

After manage to run it you will need some sniffer/proxy to monitor the requests. I have tried it with WebScarab but there is another one here and this one here

  • Run WebGoat in tomcat
  • Run you proxy/sniffer/pluging or what ever monitoring tool.
  • Start your browser
  • Setup your browser to use the proxy in case you choose to use proxy
  • Redirect your browser to WebGoat application (http://127.0.0.1:8080/WebGoat/attack )

You are ready to pass all the tests. Here are the options extracted from the lates WebGoat version:

  • Admin Functions
  • General
  • Code Quality
  • Concurrency
  • Unvalidated Parameters
  • Access Control Flaws
  • Authentication Flaws
  • Session Management Flaws
  • Cross-Site Scripting (XSS)
    • Phishing with XSS
    • LAB: Cross Site Scripting
    • Stage 1: Stored XSS
    • Stage 2: Block Stored XSS using Input Validation
    • Stage 3: Stored XSS Revisited
    • Stage 4: Block Stored XSS using Output Encoding
    • Stage 5: Reflected XSS
    • Stage 6: Block Reflected XSS
    • Stored XSS Attacks
    • Cross Site Request Forgery (CSRF)
    • Reflected XSS Attacks
    • HTTPOnly Test
    • Cross Site Tracing (XST) Attacks
  • Buffer Overflows
  • Injection Flaws
    • Command Injection
    • Blind SQL Injection
    • Numeric SQL Injection
    • Log Spoofing
    • XPATH Injection
    • LAB: SQL Injection
    • Stage 1: String SQL Injection
    • Stage 2: Parameterized Query #1
    • Stage 3: Numeric SQL Injection
    • Stage 4: Parameterized Query #2
    • String SQL Injection
    • Database Backdoors
  • Improper Error Handling
  • Insecure Storage
  • Denial of Service
  • Insecure Configuration
  • Web Services
  • AJAX Security
  • Challenge

Prevent DOS attacks with apache2

The apache module that will do the job is called mod_dosevasive. The installation is easy.

Configure the apache2

Edit your httpd.conf (usually located in /usr/local/apache/conf/httpd.conf):
Add:

<IfModule mod_dosevasive20.c>
  DOSHashTableSize 3097
  DOSPageCount 2
  DOSSiteCount 50
  DOSPageInterval 1
  DOSSiteInterval 1
  DOSBlockingPeriod 10
  DOSEmailNotify someuser@somedomain.com
  DOSSystemCommand "/dir/command  %s"
</IfModule>


Then you can restart

  • /etc/init.d/apache2 restart

Be carefull because the options are extremly important. For example if your web application use one file as a controller all the requests will go to this file you will got very often such errors:

client denied by server configuration

Here is a short help on the configuration options

- DOSHashTableSize: is the size of the table of URL and IP combined
- DOSPageCount: is the number of same page requests from the same IP during an interval that will cause that IP to be added to the block list.
- DOSSiteCount: is the number of pages requested of a site by the same IP during an interval which will cause the IP to be added to the block list.
- DOSPageInterval: is the interval that the hash table for IPs and URLs is erased (in seconds)
- DOSSiteInterval: is the intervale that the hash table of IPs is erased (in seconds)
- DOSBlockingPeriod: is the time the IP is blacked (in seconds)
- DOSEmailNotify: can be used to notify by sending an email everytime an IP is blocked
- DOSSystemCommand: is the command used to execute a command when an IP is blocked. It can be used to add a block the user from a firewall or router.
- DOSWhiteList: can be used to whitelist IPs such as 127.0.0.1

© 2026 Ivo Bardarov

Theme by Anders NorénUp ↑