Wednesday, March 31, 2010

vBulletin session table and Hash index in MEMORY Engine

In vBulletin Board System there is session table contains online user information and tracking, in which forum now, what the current URL now, So it is have huge concurrent update statement, By default this table is MEMORY engine because its data not important in case MySQL restart.

I notice early loaded in MySQL and server, So I do full tuning for the server then analyze slow query using mysqlsla and notice this query in the top:
SELECT user.username, (user.options & 512) AS invisible, user.usergroupid,
   session.userid, session.inforum, session.lastactivity, session.badlocation,
   IF(displaygroupid=0, user.usergroupid, displaygroupid) AS displaygroupid, infractiongroupid
  FROM session AS session force index(lastact_user)
  LEFT JOIN user AS user ON(user.userid = session.userid)
  WHERE session.lastactivity > 1269376225
  ORDER BY lastactivity DESC
and this is create table statement
mysql> show create table session\G
*************************** 1. row ***************************
       Table: session
Create Table: CREATE TABLE `session` (
  `sessionhash` char(32) NOT NULL DEFAULT '',
  `userid` int(10) unsigned NOT NULL DEFAULT '0',
  `host` char(15) NOT NULL DEFAULT '',
  `idhash` char(32) NOT NULL DEFAULT '',
  `lastactivity` int(10) unsigned NOT NULL DEFAULT '0',
  `location` char(255) NOT NULL DEFAULT '',
  `useragent` char(100) NOT NULL DEFAULT '',
  `styleid` smallint(5) unsigned NOT NULL DEFAULT '0',
  `languageid` smallint(5) unsigned NOT NULL DEFAULT '0',
  `loggedin` smallint(5) unsigned NOT NULL DEFAULT '0',
  `inforum` smallint(5) unsigned NOT NULL DEFAULT '0',
  `inthread` int(10) unsigned NOT NULL DEFAULT '0',
  `incalendar` smallint(5) unsigned NOT NULL DEFAULT '0',
  `badlocation` smallint(5) unsigned NOT NULL DEFAULT '0',
  `bypass` tinyint(4) NOT NULL DEFAULT '0',
  `profileupdate` smallint(5) unsigned NOT NULL DEFAULT '0',
  PRIMARY KEY (`sessionhash`),
  KEY `userid` (`userid`),
  KEY `lastact_user` (`lastactivity`,`userid`),
  KEY `idx_host_user` (`host`,`userid`),
  KEY `idhash_idx` (`idhash`),
  KEY `idx_lastactivity` (`lastactivity`)
) ENGINE=MEMORY DEFAULT CHARSET=latin1 PACK_KEYS=0


so for first impression this query use index for where and order by, so its optimal!!!!!

Unfortunately, we see it not using the index even when force it.
mysql> explain select * from session where lastactivity > 1269509561;
+----+-------------+---------+------+-------------------------------+------+---------+------+------+-------------+
| id | select_type | table   | type | possible_keys                 | key  | key_len | ref  | rows | Extra       |
+----+-------------+---------+------+-------------------------------+------+---------+------+------+-------------+
|  1 | SIMPLE      | session | ALL  | lastact_user,idx_lastactivity | NULL | NULL    | NULL | 3199 | Using where |
+----+-------------+---------+------+-------------------------------+------+---------+------+------+-------------+

mysql> explain select * from session force index (idx_lastactivity) where lastactivity > 1269509561;
+----+-------------+---------+------+------------------+------+---------+------+------+-------------+
| id | select_type | table   | type | possible_keys    | key  | key_len | ref  | rows | Extra       |
+----+-------------+---------+------+------------------+------+---------+------+------+-------------+
|  1 | SIMPLE      | session | ALL  | idx_lastactivity | NULL | NULL    | NULL | 3199 | Using where |
+----+-------------+---------+------+------------------+------+---------+------+------+-------------+
Why?

MEMORY engine support B-tree and Hash indexes, but by default use Hash algorithm:
    *      They are used only for equality comparisons that use the = or <=> operators (but are very fast). They are not used for comparison operators such as < that find a range of values.
    *      The optimizer cannot use a hash index to speed up ORDER BY operations. (This type of index cannot be used to search for the next entry in order.)
    *      MySQL cannot determine approximately how many rows there are between two values (this is used by the range optimizer to decide which index to use). This may affect some queries if you change a MyISAM table to a hash-indexed MEMORY table.
    *      Only whole keys can be used to search for a row. (With a B-tree index, any leftmost prefix of the key can be used to find rows.)

so the optimizer will not use this index.

when we alter index algorithm by:

alter table session drop index idx_lastactivity, add index `idx_lastactivity` (`lastactivity`) USING BTREE;

we notice:
mysql> explain select * from session where lastactivity > 1269509561;
+----+-------------+---------+-------+-------------------------------+------------------+---------+------+------+-------------+
| id | select_type | table   | type  | possible_keys                 | key              | key_len | ref  | rows | Extra       |
+----+-------------+---------+-------+-------------------------------+------------------+---------+------+------+-------------+
|  1 | SIMPLE      | session | range | lastact_user,idx_lastactivity | idx_lastactivity | 4       | NULL | 1919 | Using where |
+----+-------------+---------+-------+-------------------------------+------------------+---------+------+------+-------------+

mysql> explain select * from session force index (idx_lastactivity) where lastactivity > 1269509561;
+----+-------------+---------+-------+------------------+------------------+---------+------+------+-------------+
| id | select_type | table   | type  | possible_keys    | key              | key_len | ref  | rows | Extra       |
+----+-------------+---------+-------+------------------+------------------+---------+------+------+-------------+
|  1 | SIMPLE      | session | range | idx_lastactivity | idx_lastactivity | 4       | NULL | 1919 | Using where |
+----+-------------+---------+-------+------------------+------------------+---------+------+------+-------------+

So we should be careful when use MEMORY table with indexes

Wednesday, March 17, 2010

Is MySQL support Julian Dates?

I use it in Oracle and notice there are 10 days missed, for example:

ORCL> select to_date('4/10/1582','dd/mm/yyyy') SHOW_DATE from dual

SHOW_DATE
--------------
04/10/1582

ORCL> select to_date('4/10/1582','dd/mm/yyyy') + 1 SHOW_DATE from dual

SHOW_DATE
--------------
15/10/1582

Say What? the date after 4/10/1582 is 15/10/1582.

But in MySQL i try it but i didn't see this case, example:


mysql> select adddate('1582-10-04', interval 0 day);
+---------------------------------------+
| adddate('1582-10-04', interval 0 day) |
+---------------------------------------+
| 1582-10-04                            |
+---------------------------------------+
1 row in set (0.00 sec)

mysql> select adddate('1582-10-04', interval 1 day);
+---------------------------------------+
| adddate('1582-10-04', interval 1 day) |
+---------------------------------------+
| 1582-10-05                            |
+---------------------------------------+
1 row in set (0.00 sec)

Ohhh its not like Oracle!!!


So Why? Is MySQL support Julian dates?

Tuesday, February 23, 2010

Can I use latin1 to store utf8 data?

I've table contains text column and its charset is latin1, and i can store Arabic text ( and non English character) in this column and retrieve it, i don't know how is it?

So how is that? and why I need utf8?

CREATE TABLE `post` (
`postid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`threadid` int(10) unsigned NOT NULL DEFAULT '0',
`parentid` int(10) unsigned NOT NULL DEFAULT '0',
`username` varchar(100) NOT NULL DEFAULT '',
`userid` int(10) unsigned NOT NULL DEFAULT '0',
`title` varchar(250) NOT NULL DEFAULT '',
`dateline` int(10) unsigned NOT NULL DEFAULT '0',
`pagetext` mediumtext NOT NULL,
`allowsmilie` smallint(6) NOT NULL DEFAULT '0',
`showsignature` smallint(6) NOT NULL DEFAULT '0',
`ipaddress` varchar(15) NOT NULL DEFAULT '',
`iconid` smallint(5) unsigned NOT NULL DEFAULT '0',
`visible` smallint(6) NOT NULL DEFAULT '0',
`attach` smallint(5) unsigned NOT NULL DEFAULT '0',
`importthreadid` bigint(20) NOT NULL DEFAULT '0',
`importpostid` bigint(20) NOT NULL DEFAULT '0',
`infraction` smallint(5) unsigned NOT NULL DEFAULT '0',
`reportthreadid` int(10) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`postid`),
KEY `userid` (`userid`),
KEY `threadid` (`threadid`,`userid`),
KEY `datline_idx` (`dateline`),
KEY `threadid_date` (`threadid`,`dateline`),
FULLTEXT KEY `title` (`title`,`pagetext`)
) ENGINE=MyISAM AUTO_INCREMENT=32451742 DEFAULT CHARSET=latin1

Sunday, January 10, 2010

Pass application user id to MySQL database??

In all our application we connect to MySQL database as single db user, and we need to pass the end user id to the mysql database?

So how can we pass it?

I've idea to create plugin to add mysql variables ex. "app_userid" and set it when user login??!!

but i don't have any idea how to create plugin?

Or do you have another idea??

Note: i don't want to change the application code, and i need it portable.

Wednesday, January 06, 2010

First look to Chromium, Google Chrome OS

I download the binary version of Chrome OS from here, and plug it to VMware.

New thing:

the OS login is attach with Google account, so you cannot login if you havn't Google account or no internet connection, and i don't know how you can configure the internet connection.
Its very simple and the UI like Google Chrome browser.


User interface

Design goals for Google Chrome OS's user interface include using minimal screen space by combining applications and standard Web pages into a single tab strip, rather than separating the two. Designers are considering a reduced window management scheme that would operate only in full-screen mode. Secondary tasks would be handled with "panels": floating windows that dock to the bottom of the screen for tasks like chat and music players. Split screens are also under consideration for viewing two pieces of content side-by-side. Google Chrome OS will follow the Chrome browser's practice of leveraging HTML5's offline modes, background processing, and notifications. Designers propose using search and pinned tabs as a way to quickly locate and access applications.

Architecture

In preliminary design documents for the Chromium OS open source project, Google describes a three-tier architecture: firmware, browser and window manager, and system-level software and userland services.

* The firmware contributes to fast boot time by not probing for hardware, such as floppy disk drives, that are no longer common on computers, especially netbooks. The firmware also contributes to security by verifying each step in the boot process and incorporating system recovery.
* System-level software includes the Linux kernel that has been patched to improve boot performance. Userland software has been trimmed to essentials, with management by Upstart, which can launch services in parallel, re-spawn crashed jobs, and defer services in the interest of faster booting.
* The window manager handles user interaction with multiple client windows much like other X window managers.

Hardware support

Google Chrome OS is initially intended for secondary devices like netbooks, not a user's primary PC,[9] and will run on hardware incorporating an x86 or ARM.
While Chrome OS will support hard disk drives, Google has requested that its hardware partners use solid state drives due to their higher performance and reliability, as well as the lower capacity requirements inherent in an operating system that accesses applications and most user data on remote servers. Google Chrome OS consumes one-sixtieth as much drive space as Windows 7.

Companies developing hardware for the operating system include Hewlett-Packard, Acer, Adobe, Asus, Lenovo, Qualcomm, Texas Instruments, Freescale and Intel.

In December 2009, Michael Arrington of TechCrunch reported that Google has approached at least one hardware manufacturer about building a Google-branded Chrome OS netbook. According to Arrington's sources, the devices could possibly be configured for mobile broadband and be subsidized by one or more carriers.

Market implications

When Google announced the Chrome browser in September 2008 it was viewed as a continuation of the battle between Google and Microsoft ("the two giants of the digital revolution"). As of December 2009, Microsoft dominates the usage share of desktop operating systems and the software market in word processing and spreadsheet applications. The operating system dominance may be challenged directly by Google Chrome OS, and the application dominance indirectly through a shift to cloud computing. According to an analysis by PC World, Google Chrome OS represents the next step in this battle.

In November 2009 Glyn Moody writing for the Linux Journal predicted that Google's market model for the Chrome OS will be to give the software and the netbook hardware that it will run on away for free, as a means of expanding its advertising-based model. He said: "The unexpected success of netbooks over the last two years shows there is a market for this new kind of computing; giving away systems for free would take it to the next level. Then, gradually, that instant-on, secure, secondary netbook might become the one you spend most time on, and Google's ad revenues would climb even higher...."

Relationship to Android

The successive introductions of Android and Google Chrome OS, both open source, client-based operating systems, have created some market confusion, especially with Android's growing success. Microsoft CEO Steve Ballmer accused his competitor of not being able to make up its mind. Google has downplayed this conflict, suggesting that the two operating systems address different markets, mobile and personal computing, which remain distinct despite the growing convergence of the devices. Co-founder Sergey Brin suggested that the two systems "will likely converge over time".

source: wikipedia


leave you with the pictures.





Sunday, January 03, 2010

Help MySQL to still free

There's a campaign started by Monty Widenius to save MySQL from the evil clutches of Oracle. You can read about it here.


http://helpmysql.org