A memcached-like server in Ruby

A memcached-like server in Ruby - feasible?

17 Message(s) by 8 Author(s) originally posted in ruby programming

From: Tom Machinski

Date: Saturday, October 27, 2007

Hi group,

I'm run

<operating system, programming> The process of carrying out
	the {instructions} in a computer program by a computer.

	See also {dry run}.

	(1996-05-13)

ning a very high-load website

<World-Wide Web> (Or "website") Any computer on the
	{Internet} running a {World-Wide Web} {server} process.  A
	particular website is usually identified by the {hostname}
	part of a {URL}.  Multiple hostnames may actually map to the
	same computer in which case they are known as "{virtual
	servers}".

	(2005-07-12)

done in Rails.

The number and duration of queries per-page is killing us. So we're
thinking of using a caching

<memory management> /kash/ A small fast memory holding
	recently accessed data, designed to speed up subsequent access
	to the same data.  Most often applied to processor-memory
	access but also used for a local copy of data accessible over
	a network etc.

	When data is read from, or written to, {main memory} a copy is
	also saved in the cache, along with the associated main memory
	address.  The cache monitors addresses of subsequent reads to
	see if the required data is already in the cache.  If it is (a
	{cache hit}) then it is returned immediately and the main
	memory read is aborted (or not started).  If the data is not
	cached (a {cache miss}) then it is fetched from main memory
	and also saved in the cache.

	The cache is built from faster memory chips than main memory
	so a cache hit takes much less time to complete than a normal
	memory access.  The cache may be located on the same
	{integrated circuit} as the {CPU}, in order to further reduce
	the access time.  In this case it is often known as {primary
	cache} since there may be a larger, slower {secondary cache}
	outside the CPU chip.

	The most important characteristic of a cache is its {hit rate}
	- the fraction of all memory accesses which are satisfied from
	the cache.  This in turn depends on the cache design but
	mostly on its size relative to the main memory.  The size is
	limited by the cost of fast memory chips.

	The hit rate also depends on the access pattern of the
	particular program being run (the sequence of addresses being
	read and written).  Caches rely on two properties of the
	access patterns of most programs: temporal locality - if
	something is accessed once, it is likely to be accessed again
	soon, and spatial locality - if one memory location is
	accessed then nearby memory locations are also likely to be
	accessed.  In order to exploit spatial locality, caches often
	operate on several words at a time, a "{cache line}" or "cache
	block".  Main memory reads and writes are whole {cache lines}.

	When the processor wants to write to main memory, the data is
	first written to the cache on the assumption that the
	processor will probably read it again soon.  Various different
	policies are used.  In a {write-through} cache, data is
	written to main memory at the same time as it is cached.  In a
	{write-back} cache it is only written to main memory when it
	is forced out of the cache.

	If all accesses were writes then, with a write-through policy,
	every write to the cache would necessitate a main memory
	write, thus slowing the system down to main memory speed.
	However, statistically, most accesses are reads and most of
	these will be satisfied from the cache.  Write-through is
	simpler than write-back because an entry that is to be
	replaced can just be overwritten in the cache as it will
	already have been copied to main memory whereas write-back
	requires the cache to initiate a main memory write of the
	flushed entry followed (for a processor read) by a main memory
	read.  However, write-back is more efficient because an entry
	may be written many times in the cache without a main memory
	access.

	When the cache is full and it is desired to cache another line
	of data then a cache entry is selected to be written back to
	main memory or "flushed".  The new line is then put in its
	place.  Which entry is chosen to be flushed is determined by a
	"{replacement algorithm}".

	Some processors have separate instruction and data caches.
	Both can be active at the same time, allowing an instruction
	fetch to overlap with a data read or write.  This separation
	also avoids the possibility of bad {cache conflict} between
	say the instructions in a loop and some data in an array which
	is accessed by that loop.

	See also {direct mapped cache}, {fully associative cache},
	{sector mapping}, {set associative cache}.

	(1997-06-25)

layer

<networking> The {software} and/or {hardware} environment of
	two or more communications devices or computers in which a
	particular {network} {protocol} operates.  A network
	connection may be thought of as a set of more or less
	independent protocols, each in a different layer or level.
	The lowest layer governs direct host-to-host communication
	between the hardware at different hosts; the highest consists
	of user {application programs}.  Each layer uses the layer
	beneath it and provides a service for the layer above.  Each
	networking component {hardware or software} on one host uses
	{protocols} appropriate to its layer to communicate with the
	corresponding component (its "peer") on another host.  Such
	layered protocols are sometimes known as peer-to-peer
	protocols.

	The advantages of layered {protocols} is that the methods of
	passing information from one layer to another are specified
	clearly as part of the {protocol} suite, and changes within a
	protocol layer are prevented from affecting the other layers.
	This greatly simplifies the task of designing and maintaining
	communication systems.

	Examples of layered protocols are {TCP/IP}'s five layer
	{protocol stack} and the {OSI} seven layer model.

	(1997-05-05)

like memcached. Except we'd like
something more sophisticated than memcached.

Allow me to explain.

memcached is like an object

<object-oriented> In {object-oriented programming}, an
	instance of the data structure and behaviour defined by the
	object's {class}.  Each object has its own values for the
	{instance variables} of its class and can respond to the
	{methods} defined by its class.

	For example, an object of the "Point" class might have
	instance variables "x" and "y" and might respond to the "plot"
	method by drawing a dot on the screen at those coordinates.

	(2004-01-26)

, with a very limited API: basically
#get_value_by_key and #set_value_by_key.

One thing we need, that is not supported by memcached, is to be able to
store

[probably from "main store"] In some varieties of Commonwealth
	hackish, the preferred synonym for {core}.  Thus, "bringing a
	program into store" means not that one is returning
	shrink-wrapped software but that a program is being {swap}ped
	in.

	[{Jargon File}]

a large set of very large objects, and then retrieve only a few
of them by certain parameters. For example, we may want to store 100K
Foo instances, and retrieve only the first 20 - sort

1. <application, algorithm> To arrange a collection of items
	in some specified order.  The items - {records} in a file or
	data structures in memory - consist of one or more {fields} or
	members.  One of these fields is designated as the "sort key"
	which means the records will be ordered according to the value
	of that field.  Sometimes a sequence of key fields is
	specified such that if all earlier keys are equal then the
	later keys will be compared.  Within each field some ordering
	is imposed, e.g. ascending or descending numerical, {lexical
	ordering}, or date.

	Sorting is the subject of a great deal of study since it is a
	common operation which can consume a lot of computer time.
	There are many well-known sorting {algorithms} with different
	time and space behaviour and programming {complexity}.

	Examples are {quicksort}, {insertion sort}, {bubble sort},
	{heap sort}, and {tree sort}.  These employ many different
	data structures to store sorted data, such as {arrays},
	{linked lists}, and {binary trees}.

	2. <tool> The {Unix} utility program for sorting lines of
	files.

	{Unix manual page}: sort(1).

	(1997-02-12)

ed by their
#created_on attribute

<data> A named value or relationship that exists for some or
	all {instances} of some {entity} and is directly associated
	with that instance.

	Examples include the {href} attribute of an {HTML} {anchor}
	element, the columns of a database table considered as
	attributes of each row, and the {members} ({properties} and
	{methods} of an {object} in {OOP}.  This contrasts with the
	contents of some kind of container (e.g. an array), which are
	typically not named.  The contents of an {associative array},
	though they might be considered to be named by their key
	values, are not normally thought of as attributes.

	(2001-02-04)

- whose #bar attribute equal 23.

We could store all those 100K Foo instances normally on the memcached
server

1. A program which provides some service to other ({client})
	programs.  The connection between client and server is
	normally by means of {message passing}, often over a
	{network}, and uses some {protocol} to encode the client's
	requests and the server's responses.  The server may run
	continuously (as a {daemon}), waiting for requests to arrive
	or it may be invoked by some higher level daemon which
	controls a number of specific servers ({inetd} on {Unix}).

	There are many servers associated with the {Internet}, such as
	those for {HTTP}, {Network File System}, {Network Information
	Service} (NIS), {Domain Name System} (DNS), {FTP}, {news},
	{finger}, {Network Time Protocol}.  On Unix, a long list can
	be found in /etc/services or in the {NIS} database "services".
	See {client-server}.

	2. A computer which provides some service for other computers
	connected to it via a network.  The most common example is a
	{file server} which has a local disk and services requests
	from remote clients to read and write files on that disk,
	often using {Sun}'s {Network File System} (NFS) {protocol} or
	{Novell Netware} on {PCs}.  Another common example is a {web
	server}.

	[{Jargon File}]

	(2003-12-29)

, and let the Rails process

1. <operating system, software> The sequence of states of an
	executing {program}.  A process consists of the program {code}
	(which may be shared with other processes which are executing
	the same program), private data, and the state of the
	{processor}, particularly the values in its {registers}.  It
	may have other associated resources such as a {process
	identifier}, open files, {CPU time} limits, {shared memory},
	{child processes}, and {signal handlers}.

	One process may, on some {platforms}, consist of many
	{threads}.  A {multitasking} {operating system} can run
	multiple processes {concurrently} or in {parallel}, and allows
	a process to spawn "child" processes.

	(2001-06-16)

	2. <business> The sequence of activities, people, and systems
	involved in carrying out some business or achieving some
	desired result.  E.g. software development process, project
	management process, configuration management process.

	(2001-06-16)

retrieve them on each request. Then
the process could perform the filter

1. (Originally {Unix}, now also {MS-DOS}) A program that
	processes an input data stream into an output data stream in
	some well-defined way, and does no I/O to anywhere else except
	possibly on error conditions; one designed to be used as a
	stage in a {pipeline} (see {plumbing}).  Compare {sponge}.

	2. ({functional programming}) A {higher-order function} which
	takes a {predicate} and a list and returns those elements of
	the list for which the predicate is true.  In {Haskell}:

		filter p []     = []
		filter p (x:xs) = if p x then x : rest else rest
				  where
				  rest = filter p xs

	See also {filter promotion}.

	[{Jargon File}]

ing itself. Problem is that it's
very suboptimal, because we'd have to transfer a lot of data

<data, data processing, jargon> /day't*/ (Or "raw data")
	Numbers, {characters}, {images}, or other method of recording,
	in a form which can be assessed by a human or (especially)
	input into a {computer}, stored and {processed} there, or
	transmitted on some {digital channel}.  Computers nearly
	always represent data in {binary}.

	Data on its own has no meaning, only when interpreted by some
	kind of {data processing system} does it take on meaning and
	become {information}.

	People or computers can find patterns in data to perceive
	information, and information can be used to enhance
	{knowledge}.  Since knowledge is prerequisite to wisdom, we
	always want more data and information.  But, as modern
	societies verge on {information overload}, we especially need
	better ways to find patterns.

	1234567.89 is data.

	"Your bank balance has jumped 8087% to $1234567.89" is
	information.

	"Nobody owes me that much money" is knowledge.

	"I'd better talk to the bank before I spend it, because of
	what has happened to other people" is wisdom.

	(1999-04-30)

to each
process on each request, and very little of that data is actually
needed after the processing. I.e. we'd pass 100K large objects,
while the process only real

1. Not simulated.  Often used as a specific antonym to
	{virtual} in any of its jargon senses.

	2. <mathematics> {real number}.

	[{Jargon File}]

	(1997-03-12)

ly needs 20 of them.

Ideally, we could call:

memcached_improved.fetch_newest( :attributes => { :bar => 23 }, :limit
=> 20 )

and have the improved_memcached server filter and return only the
required 20 objects by itself.

Now the question is:

How expensive'd it be to write

1. <chat> {Unix}'s simple {talk} command and {protocol}.
	write has been largely superseded by {talk} and then {irc}.

	An enhancement, {RWP}, has been proposed.

	2. <tool> A simple {text editor} for {Windows}.

	(1998-04-28)

memcached_improved?

On the surface, this might seem easy to do with something like
Daemons[1] in Ruby

<language>

	1. A {relational language} designed by Jones and M. Sheeran in
	1986 for describing and designing circuits (a {hardware
	description language}).  Ruby programs denote {binary
	relations} and programs are built-up inductively from
	primitive relations using a pre-defined set of {relational
	operators}.  Ruby programs also have a geometric
	interpretation as networks of primitive relations connected by
	wires, which is important when layout is considered in circuit
	design.

	Ruby has been continually developed since 1986, and has been
	used to design many different kinds of circuits, including
	{systolic arrays}, {butterfly networks} and arithmetic
	circuits.

	{(ftp://ftp.cs.chalmers.se/pub/misc/ruby/)}.

	E-mail: <graham@cs.chalmers.se>.

	["Ruby - A Language of Relations and Higher-Order Functions",
	M. Sheeran, Proc 3rd Banff Workshop on Hardware Verification,
	Springer 1990].

	(1994-10-27)

	2. One of five pedagogical languages based on {Markov
	algorithms}, used in Higman's report (below).  The other
	languages are {Brilliant}, {Diamond}, {Nonpareil}, and
	{Pearl}.

	["Nonpareil, a Machine Level Machine Independent Language for
	the Study of Semantics", B. Higman, ULICS Intl Report No ICSI
	170, U London (1968)].

	(1994-10-27)

	3. A fully {object oriented} {interpreted} {scripting
	language} by Yukihiro Matsumoto <matz@netlab.co.jp>.

	Similar in scope to {Perl} and {Python}, Ruby has high-level
	{data types}, automatic {memory management}, {dynamic typing},
	a {module} system, {exceptions}, and a rich standard library.
	Other features are {CLU}-style {iterators} for {loop
	abstraction}, {singleton classes}/{methods} and {lexical
	closures}.

	In Ruby, everything is an {object}, including the basic data
	types.  For example, the number 1 is an instance of {class}
	Fixnum.

	Current version (stable): 1.6.7, as of 2002-03-01.

	{Ruby Home (http://www.ruby-lang.org/)}.

	{Ruby Central (http://www.rubycentral.com/)}.

	["Programming Ruby - The Pragmatic Programmer's Guide", David
	Thomas, Andrew Hunt, Yukihiro Matsumoto pub. Addison Wesley
	2000].

	(2002-06-19)

(as most of our programmers are Rubyists). Just
write a simple class, have it run a TCP

<networking, protocol> (TCP) The most common {transport layer}
	{protocol} used on {Ethernet} and the {Internet}.  It was
	developed by {DARPA}.

	TCP is the {connection-oriented} protocol built on top of
	{Internet Protocol} (IP) and is nearly always seen in the
	combination {TCP/IP} (TCP over IP).  It adds {reliable
	communication} and {flow-control} and provides {full-duplex},
	process-to-process connections.

	TCP is defined in {STD 7} and {RFC} 793.

	{User Datagram Protocol} is the other, connectionless,
	protocol that runs on top of IP.

	(2001-06-14)

server and respond to
requests. Yet I'm sure it's not that simple, otherwise memcached'd
have been trivial to write. There are probably stability issues for
multiple concurrent clients, multiple simultaneous read/write requests
(race conditions etc.) and heavy loads.

So, what do you think:

1) How'd you app

<programming, operating system> (Or "application", "app") A
	complete, self-contained program that performs a specific
	function directly for the user.  This is in contrast to system
	software such as the {operating system} {kernel}, {server}
	processes and libraries which exists to support application
	programs.

	Editors for various kinds of documents, {spreadsheets}, and
	text formatters are common examples of applications.  Network
	applications include clients such as those for {FTP},
	{electronic mail}, {telnet} and {WWW}.

	The term is used fairly loosely, for instance, some might say
	that a client and server together form a distributed
	application, others might argue that editors and compilers
	were not applications but tools for building applications.

	One distinction between an application program and the
	operating system is that applications always run in "user
	mode" (or "non-privileged mode"), while operating systems and
	related utilities may run in "supervisor mode" (or "privileged
	mode").

	The term may also be used to distinguish programs which
	communicate via a {graphical user interface} from those which
	are executed from the {command line}.

	(1994-11-28)

roach the development

The process of analysis, design, coding and testing software.

of memcached_improved?

2) Is this task doable in Ruby? Or maybe only a Ruby + X combination
(X probably being C)?

3) How much time / effort / people / expertise should such a task
require? Is it feasible

<algorithm> A description of an {algorithm} that takes
	{polynomial} time (that is, for a problem set of size N, the
	resources required to solve the problem can be expressed as
	some polynomial involving N).

	Problems that are "feasible" are said to be "in P" where P is
	polynomial time.  Problems that are "possible" but not
	"feasible" are said to be "in NP".

	(2001-04-12)

for a smallish team (~4 programmers) to put
together as a side-project over a couple of weeks?

Thanks,
-Tom
--
[1] http://daemon

<operating system> /day'mn/ or /dee'mn/ (From the mythological
	meaning, later rationalised as the acronym "Disk And Execution
	MONitor") A program that is not invoked explicitly, but lies
	dormant waiting for some condition(s) to occur.  The idea is
	that the perpetrator of the condition need not be aware that a
	daemon is lurking (though often a program will commit an
	action only because it knows that it will implicitly invoke a
	daemon).

	For example, under {ITS} writing a file on the {LPT} spooler's
	directory would invoke the spooling daemon, which would then
	print the file.  The advantage is that programs wanting files
	printed need neither compete for access to, nor understand any
	idiosyncrasies of, the {LPT}.  They simply enter their
	implicit requests and let the daemon decide what to do with
	them.  Daemons are usually spawned automatically by the
	system, and may either live forever or be regenerated at
	intervals.

	{Unix} systems run many daemons, chiefly to handle requests
	for services from other {hosts} on a {network}.  Most of these
	are now started as required by a single real daemon, {inetd},
	rather than running continuously.  Examples are {cron} (local
	timed command execution), {rshd} (remote command execution),
	{rlogind} and {telnetd} (remote login), {ftpd}, {nfsd} (file
	transfer), {lpd} (printing).

	Daemon and {demon} are often used interchangeably, but seem to
	have distinct connotations (see {demon}).  The term "daemon"
	was introduced to computing by {CTSS} people (who pronounced
	it /dee'mon/) and used it to refer to what {ITS} called a
	{dragon}.

	[{Jargon File}]

	(1995-05-11)

s.rubyforge.org/

From: Lionel Bouton

Date: Saturday, October 27, 2007

wrote in message:

Hi group,
I'm running a very high-load website done in Rails.
The number and duration of queries per-page is killing us. So we're
thinking of using a caching layer like memcached. Except we'd like
something more sophisticated than memcached.
Allow me to explain.
memcached is like an object, with a very limited API: basically
#get_value_by_key and #set_value_by_key.
One thing we need, that is not supported by memcached, is to be able to
store a large set of very large objects, and then retrieve only a few
of them by certain parameters. For example, we may want to store 100K
Foo instances, and retrieve only the first 20 - sorted by their
#created_on attribute - whose #bar attribute equal 23.

It looks like the job

<operating system> All activities involved in completing any
	project on a computer from start to finish.  A job may involve
	several {processes} and several {programs}.

	This term originates from a time when a user would manually
	submit a job as a deck of {punched cards} which would
	typically include {source code} interspersed with {job control
	language} instructions to guide phases of the job such as
	{compilation}, {linking}, {execution} and printing.

	(2005-03-16)

a database'd do for you. Retrieving 20 large
objects with such conditions should be a piece of cake for any properly
tuned database. Did you try this with PostgreSQL

<database> /'post-gres-kyu-el/ An enhancement of the
	{POSTGRES} {database} system.

	PostgreSQL is an advanced {relational database management
	system} with some {object oriented} approaches.  PostgreSQL is
	developed and distributed as {free software}, and while
	retaining its freedom it remains technically and featurewise a
	worthy competitor to even the most advanced commercial
	alternatives.

	It was also one of the first databases to offer {MVCC} as
	opposed to {row-level locking} or {table locking}, thereby
	greatly improving multi-user performance.

	PostgreSQL implements an extended subset of {ANSI} {SQL} and
	runs on many {platforms}.  It also has {interfaces} to many
	different {programming languages} and database {protocols},
	like {ODBC} and {JDBC}.

	{Home (http://www.postgresql.org/)}.

	(1999-09-18)

or MySQL with indexes
on created_on and bar? How much memory

<storage> These days, usually used synonymously with {Random
	Access Memory} or {Read-Only Memory}, but in the general sense
	it can be any device that can hold {data} in
	{machine-readable} format.

	(1996-05-25)

did you give your database to
play with ? If the size of the objects is so bad it takes too much time
to extract from the DB (or the trafic is too much for the DB to use its
own disk

<storage> 1. {magnetic disk}.

	2. {compact disc}.

	3. {optical disk}.

	Note: the american spelling, "disk", is normal for most
	computer disks whereas "compact disc", having come to
	computers via the audio world, is correctly spelled with a
	"c", indeed, this spelling is part of the CD standard.

	(1995-07-30)

cache

<memory management> /kash/ A small fast memory holding
	recently accessed data, designed to speed up subsequent access
	to the same data.  Most often applied to processor-memory
	access but also used for a local copy of data accessible over
	a network etc.

	When data is read from, or written to, {main memory} a copy is
	also saved in the cache, along with the associated main memory
	address.  The cache monitors addresses of subsequent reads to
	see if the required data is already in the cache.  If it is (a
	{cache hit}) then it is returned immediately and the main
	memory read is aborted (or not started).  If the data is not
	cached (a {cache miss}) then it is fetched from main memory
	and also saved in the cache.

	The cache is built from faster memory chips than main memory
	so a cache hit takes much less time to complete than a normal
	memory access.  The cache may be located on the same
	{integrated circuit} as the {CPU}, in order to further reduce
	the access time.  In this case it is often known as {primary
	cache} since there may be a larger, slower {secondary cache}
	outside the CPU chip.

	The most important characteristic of a cache is its {hit rate}
	- the fraction of all memory accesses which are satisfied from
	the cache.  This in turn depends on the cache design but
	mostly on its size relative to the main memory.  The size is
	limited by the cost of fast memory chips.

	The hit rate also depends on the access pattern of the
	particular program being run (the sequence of addresses being
	read and written).  Caches rely on two properties of the
	access patterns of most programs: temporal locality - if
	something is accessed once, it is likely to be accessed again
	soon, and spatial locality - if one memory location is
	accessed then nearby memory locations are also likely to be
	accessed.  In order to exploit spatial locality, caches often
	operate on several words at a time, a "{cache line}" or "cache
	block".  Main memory reads and writes are whole {cache lines}.

	When the processor wants to write to main memory, the data is
	first written to the cache on the assumption that the
	processor will probably read it again soon.  Various different
	policies are used.  In a {write-through} cache, data is
	written to main memory at the same time as it is cached.  In a
	{write-back} cache it is only written to main memory when it
	is forced out of the cache.

	If all accesses were writes then, with a write-through policy,
	every write to the cache would necessitate a main memory
	write, thus slowing the system down to main memory speed.
	However, statistically, most accesses are reads and most of
	these will be satisfied from the cache.  Write-through is
	simpler than write-back because an entry that is to be
	replaced can just be overwritten in the cache as it will
	already have been copied to main memory whereas write-back
	requires the cache to initiate a main memory write of the
	flushed entry followed (for a processor read) by a main memory
	read.  However, write-back is more efficient because an entry
	may be written many times in the cache without a main memory
	access.

	When the cache is full and it is desired to cache another line
	of data then a cache entry is selected to be written back to
	main memory or "flushed".  The new line is then put in its
	place.  Which entry is chosen to be flushed is determined by a
	"{replacement algorithm}".

	Some processors have separate instruction and data caches.
	Both can be active at the same time, allowing an instruction
	fetch to overlap with a data read or write.  This separation
	also avoids the possibility of bad {cache conflict} between
	say the instructions in a loop and some data in an array which
	is accessed by that loop.

	See also {direct mapped cache}, {fully associative cache},
	{sector mapping}, {set associative cache}.

	(1997-06-25)

efficiently) you could only retrieve the ids in the first
pass with hand-crafted SQL

<language, database, standard> /S Q L/ An industry-standard
	language for creating, updating and, querying {relational
	database management systems}.

	SQL was developed by {IBM} in the 1970s for use in {System R}.
	It is the {de facto standard} as well as being an {ISO} and
	{ANSI} {standard}.  It is often embedded in general purpose
	programming languages.

	The first SQL standard, in 1986, provided basic language
	constructs for defining and manipulating {tables} of data; a
	revision in 1989 added language extensions for {referential
	integrity} and generalised {integrity} {constraints}.  Another
	revision in 1992 provided facilities for {schema} manipulation
	and {data administration}, as well as substantial enhancements
	for data definition and data manipulation.

	Development is currently underway to enhance SQL into a
	computationally complete language for the definition and
	management of {persistent}, complex objects.  This includes:
	generalisation and specialisation hierarchies, {multiple
	inheritance}, user defined {data types}, {triggers} and
	{assertions}, support for {knowledge based systems},
	{recursive query expressions}, and additional data
	administration tools.  It also includes the specification of
	{abstract data types} (ADTs), object identifiers, {methods},
	{inheritance}, {polymorphism}, {encapsulation}, and all of the
	other facilities normally associated with object data
	management.

	The emerging {SQL3} standard is expected to be complete in
	1998.

	According to Allen G. Taylor, SQL does _not_ stand for
	"Structured Query Language".  That, like "SEQUEL" (and its
	pronunciation /see'kw*l/), was just another unofficial name
	for a precursor of SQL.  However, the IBM SQL Reference manual
	for DB2 and Craig Mullins's "DB2 Developer's Guide" say SQL
	_does_ stand for "Structured Query Language".

	{SQL Standards (http://www.jcc.com/sql_stnd.html)}.

	{An SQL parser
	(ftp://ftp.ora.com/published/oreilly/nutshell/lexyacc/)} is
	described in "Lex & Yacc", by Levine, Mason & Brown published
	by O'Reilly.

	{The 1995 SQL Reunion: People, Projects, and Politics
	(http://www.mcjones.org/System_R/SQL_Reunion_95/)}.

	["A Guide to the SQL Standard", C.J. Date, A-W 1987].

	["SQL for Dummies", Allen G. Taylor, IDG Books Worldwide].

	(2005-11-17)

and then fetch the whole objects using
memcache (and only go to the DB if memcache does not have the object you
are looking for).

Lionel.

From: Tom Machinski

Date: Saturday, October 27, 2007

wrote in message:

It looks like the job a database'd do for you. Retrieving 20 large
objects with such conditions should be a piece of cake for any properly
tuned database. Did you try this with PostgreSQL or MySQL with indexes
on created_on and bar?

Yes, I'm using MySQL 5, and all query

1. <database, information science> A user's (or {agent}'s)
	request for information, generally as a formal request to a
	{database} or {search engine}.

	{SQL} is the most common {database query language}.

	2. <character> {question mark}.

	(1997-04-09)

columns are indexed.

How much memory did you give your database to
play with?

Not sure right now, I will ask my admin

{system administration}

and reply.

If the size of the objects is so bad it takes too much time
to extract from the DB (or the trafic is too much for the DB to use its
own disk cache efficiently) you could only retrieve the ids in the first
pass with hand-crafted SQL and then fetch the whole objects using
memcache (and only go to the DB if memcache does not have the object you
are looking for).

Might be a good idea.

Lionel.

Long term, my goal

<programming> In {logic programming}, a {predicate} applied to
	its {arguments} which the system attempts to prove by matching
	it against the {clauses} of the program.  A goal may fail or
	it may succeed in one or more ways.

	(1997-07-14)

is to minimize the amount of queries that hit

1. <architecture> {cache hit}.

	2. <World-Wide Web> A request to a {web server} from a {web
	browser} or other {client} (e.g. a {robot}).

	The number of hits on a server may be important for
	determining advertising revenue.

	In the course of loading a single {web page}, a browser may
	hit a web server many times e.g. to retrieve the page itself
	and each {image} on the page.  In contrast, caching by
	browsers and {web proxies} reduces the number of hits on the
	server because some requests are satisfied from the cache.

	3. <jargon> To press and release a key on the keyboard.  Some
	prefer the less aggressive "tap".

	(2000-02-20)

the
database. Some of the queries are more complex than the relatively
simple example I have given here. And I do not think I could optimize
them much beyond 0.01 secs per query.

I was hoping to alleviate with memcached_improved some of the pains
associated with database scaling, e.g. building a replicating cluster
etc. Basically what memcached does for you, except as demonstrated,
memcached by itself seems insufficient for our needs.

Thanks,
-Tom

From: ara.t.howard

Date: Sunday, October 28, 2007

wrote in message:

Hi group,
I'm running a very high-load website done in Rails.
The number and duration of queries per-page is killing us. So we're
thinking of using a caching layer like memcached. Except we'd like
something more sophisticated than memcached.
Allow me to explain.
memcached is like an object, with a very limited API: basically
#get_value_by_key and #set_value_by_key.
One thing we need, that is not supported by memcached, is to be able to
store a large set of very large objects, and then retrieve only a few
of them by certain parameters. For example, we may want to store 100K
Foo instances, and retrieve only the first 20 - sorted by their
#created_on attribute - whose #bar attribute equal 23.
We could store all those 100K Foo instances normally on the memcached
server, and let the Rails process retrieve them on each request. Then
the process could perform the filtering itself. Problem is that it's
very suboptimal, because we'd have to transfer a lot of data to each
process on each request, and very little of that data is actually
needed after the processing. I.e. we'd pass 100K large objects,
while the process only really needs 20 of them.
<snip>

i'm reading this as

- need query
- need readonly
- need sorting
- need fast
- need server

and thinking: how is not this a readonly slave database? I think that
mysql can either do this with a readonly slave *or* it cannot be done
with modest resources.

my 2cts.
a @xxxxxxxxxxx http://codeforpeople.com/
--
it isn't enough to be compassionate. you must act.
h.h. the 14th dalai lama

From: M. Edward

Date: Sunday, October 28, 2007

wrote in message:
wrote in message:
Hi group,

I'm running a very high-load website done in Rails.

The number and duration of queries per-page is killing us. So we're
thinking of using a caching layer like memcached. Except we'd like
something more sophisticated than memcached.

Allow me to explain.

memcached is like an object, with a very limited API: basically
#get_value_by_key and #set_value_by_key.

One thing we need, that is not supported by memcached, is to be able to
store a large set of very large objects, and then retrieve only a few
of them by certain parameters. For example, we may want to store 100K
Foo instances, and retrieve only the first 20 - sorted by their
#created_on attribute - whose #bar attribute equal 23.

We could store all those 100K Foo instances normally on the memcached
server, and let the Rails process retrieve them on each request. Then
the process could perform the filtering itself. Problem is that it's
very suboptimal, because we'd have to transfer a lot of data to each
process on each request, and very little of that data is actually
needed after the processing. I.e. we'd pass 100K large objects,
while the process only really needs 20 of them.
<snip>
i'm reading this as
- need query
- need readonly
- need sorting
- need fast
- need server
and thinking: how is not this a readonly slave database? I think that
mysql can either do this with a readonly slave *or* it cannot be done
with modest resources.
my 2cts.

Add "large set of very large (binary?) objects". So ... yes, at least
*one* database/server. This is exactly the sort of thing you *can* throw
hardware at. I guess I'd pick PostgreSQL over MySQL for something like
that, but unless you're a billionaire, I'd be doing it from disk and not
from RAM

1. <storage> {Random Access Memory}.

	2. <humour> Rarely Adequate Memory.

	A humorous reference to the fact that programs and data
	expand to fill the memory available.

	(1995-04-22)

. RAM-based "databases" look really attractive on paper, but
they tend to look better than they really are for a lot of reasons:

1. *Good* RAM -- the kind that does not fall over in a ragged heap

1. <programming> An area of memory used for {dynamic memory
	allocation} where blocks of memory are allocated and freed in
	an arbitrary order and the pattern of allocation and size of
	blocks is not known until {run time}.  Typically, a program
	has one heap which it may use for several different purposes.

	Heap is required by languages in which functions can return
	arbitrary data structures or functions with {free variables}
	(see {closure}).  In {C} functions {malloc} and {free} provide
	access to the heap.

	Contrast {stack}.  See also {dangling pointer}.

	2. <programming> A data structure with its elements partially
	ordered (sorted) such that finding either the minimum or the
	maximum (but not both) of the elements is computationally
	inexpensive (independent of the number of elements), while
	both adding a new item and finding each subsequent
	smallest/largest element can be done in O(log n) time, where n
	is the number of elements.

	Formally, a heap is a {binary tree} with a key in each {node},
	such that all the {leaves} of the tree are on two adjacent
	levels; all leaves on the lowest level occur to the left and
	all levels, except possibly the lowest, are filled; and the
	key in the {root} is at least as large as the keys in its
	children (if any), and the left and right subtrees (if they
	exist) are again heaps.

	Note that the last condition assumes that the goal is finding
	the minimum quickly.

	Heaps are often implemented as one-dimensional {arrays}.
	Still assuming that the goal is finding the minimum quickly
	the {invariant} is

	   heap[i] <= heap[2*i] and heap[i] <= heap[2*i+1] for all i,

	where heap[i] denotes the i-th element, heap[1] being the
	first.  Heaps can be used to implement {priority queues} or in
	{sort} algorithms.

	(1996-02-26)

when
challenged with "memtest86" -- isn't inexpensive. Let's say the objects
are "very large" -- how about a typical CD length of 700 MB? OK ... too
big -- how about a three minute video highly compressed. How big are
those puppies? Let's assume a megabyte. 100K of those is 100 GB. Wanna
price 100 GB of *good* RAM? Even with compression, it does not take much
stuff to fill up a 160 GB iPod, right?

2. A good RDBMS

<database> (RDBMS - relational database management system) A
	{database} based on the {relational model} developed by
	{E.F. Codd}.  A relational database allows the definition of
	data structures, storage and retrieval operations and
	{integrity constraints}.  In such a database the data and
	relations between them are organised in {tables}.  A table is
	a collection of rows or {records} and each row in a table
	contains the same {fields}.  Certain fields may be designated
	as {keys}, which means that searches for specific values of
	that field will use indexing to speed them up.

	Where fields in two different tables take values from the same
	set, a {join} operation can be performed to select related
	records in the two tables by matching values in those fields.
	Often, but not always, the fields will have the same name in
	both tables.  For example, an "orders" table might contain
	(customer_id, product_code) pairs and a "products" table might
	contain (product_code, price) pairs so to calculate a given
	customer's bill you would sum the prices of all products
	ordered by that customer by joining on the product-code fields
	of the two tables.  This can be extended to joining multiple
	tables on multiple fields.  Because these relationships are
	only specified at retreival time, relational databases are
	classed as {dynamic database management system}.

	The first commercial RDBMS was the {Multics Relational Data
	Store}, first sold in 1978.

	{INGRES}, {Oracle}, {Sybase, Inc.}, {Microsoft Access}, and
	{Microsoft SQL Server} are well-known database products and
	companies.  Others include {PostgreSQL}, {SQL/DS}, and {RDB}.

	["Managing Data Bases, Four Critical Factors" Michael
	M. Gorman, QED Information Sciences, Inc.].

	["An Introduction To Database Systems" (6th ed) C. J. Date,
	Addison Wesley (an excellent source of detailed info)].

	["An End-User's Guide to Data Base" James Martin, Prentice
	Hall (excellent place to begin learning about DBMS)].

	(2002-06-10)

design

<process> The approach that engineering (and some other)
	disciplines use to specify how to create or do something.  A
	successful design must satisfies a (perhaps informal)
	{functional specification} (do what it was designed to do);
	conforms to the limitations of the target medium (it is
	possible to implement); meets implicit or explicit
	requirements on performance and resource usage (it is
	efficient enough).

	A design may also have to satisfy restrictions on the design
	process itself, such as its length or cost, or the tools
	available for doing the design.

	In the {software life-cycle}, design follows {requirements
	analysis} and is followed by implementation.

	["Object-Oriented Analysis and Design with Applications", 2nd
	ed., Grady Booch].

	(1996-12-08)

/ query planner is amazingly intelligent, and you
can give it hints. It might take you a couple of weeks to build your
indexes but your queries will run fast afterwards.

3. RAID

<storage, architecture> (RAID.  Originally "Redundant Arrays
	of Inexpensive Disks") A project at the computer science
	department of the {University of California at Berkeley},
	under the direction of Professor Katz, in conjunction with
	Professor {John Ousterhout} and Professor {David Patterson}.

	The project is reaching its culmination with the
	implementation of a prototype disk array file server with a
	capacity of 40 GBytes and a sustained bandwidth of 80
	MBytes/second.  The server is being interfaced to a 1 Gb/s
	{local area network}.  A new initiative, which is part of the
	{Sequoia 2000} Project, seeks to construct a geographically
	distributed storage system spanning disk arrays and automated
	libraries of {optical disks} and tapes.  The project will
	extend the interleaved storage techniques so successfully
	applied to disks to tertiary storage devices.  A key element
	of the research will be to develop techniques for managing
	latency in the I/O and network paths.

	The original ("..Inexpensive..") term referred to the 3.5 and
	5.25 inch disks used for the first RAID system but no longer
	applies.

	The following standard RAID specifications exist:

	 RAID 0	Non-redundant striped array
	 RAID 1	Mirrored arrays
	 RAID 2	Parallel array with ECC
	 RAID 3	Parallel array with parity
	 RAID 4	Striped array with parity
	 RAID 5	Striped array with rotating parity

	{(ftp://wuarchive.wustl.edu/doc/techreports/berkeley.edu/raid/raidPapers)}.
	{(http://HTTP.CS.Berkeley.EDU/projects/parallel/research_summaries/14-Computer-Architecture/)}.

	["A Case for Redundant Arrays of Inexpensive Disks (RAID)",
	"D. A. Patterson and G. Gibson and R. H. Katz", Proc ACM
	SIGMOD Conf, Chicago, IL, Jun 1988].

	["Introduction to Redundant Arrays of Inexpensive Disks
	(RAID)", "D. A. Patterson and P. Chen and G. Gibson and
	R. H. Katz", IEEE COMPCON 89, San Francisco, Feb-Mar 1989].

	(1995-07-20)

10 is your friend. Mirroring preserves your data when a disk
dies, and striping makes it come into RAM quickly.

4. Enterprise-grade SAN

s have lots of buffering built in. And for that
stuff, you do not have to be a billionaire -- just a plain old millionaire.

"Premature optimization is the root

1. <operating system> The {Unix} {superuser} account (with
	user name "root" and user ID 0) that overrides file
	permissions.  The term {avatar} is also used.  By extension,
	the privileged system-maintenance login on any {operating
	system}.

	See {root mode}, {go root}, {wheel}.

	[{Jargon File}]

	(1994-10-27)

	2. <operating system> {root directory}.

	(1996-11-21)

	3. <data> {root node}.

	(1998-11-14)

of all evil?" Bullshit! :)

From: ara.t.howard

Date: Sunday, October 28, 2007

wrote in message:

it does not take much stuff to fill up a 160 GB iPod, right?

http://drawohara.tumblr.com/post/17471102

could not resist...

cheers.

a @xxxxxxxxxxx http://codeforpeople.com/
--
share your knowledge. it's a way to achieve immortality.
h.h. the 14th dalai lama

From: Bill Kelly

Date: Sunday, October 28, 2007

From: "ara.t.howard" <ara.t.howard@xxxxxxxxxxx>
wrote in message:
it does not take much stuff to fill up a 160 GB iPod, right?
http://drawohara.tumblr.com/post/17471102

BTW, my wife and I were only able to fit about 3/5ths of our CD
collection on our 40 gig

<unit> 2^30 = 1,073,741,824 {bytes} = 1024 {megabytes}.

	Roughly the amount of data required to encode a human gene
	sequence (including all the redundant codons).

	See {prefix}.

	(1995-09-29)

iPod. (I rip at 320kbps mp3 admittedly.)

So while a 160 GB iPod'd be slightly overkill for us, it
would not be outrageously so.Regards,

Bill

From: Tom Machinski

Date: Sunday, October 28, 2007

wrote in message:

The other thing you can play with is using sqlite as the local (one
per app server) cache engine.

Thanks, but if I'm already caching at the local process level, I might
as well cache to in-memory Ruby objects; the entire data-set is not
that huge for a high-end server RAM capacity: about 500 MB all in all.

YS.

-Tom

From: Tom Machinski

Date: Sunday, October 28, 2007

wrote in message:

i'm reading this as
- need query
- need readonly
- need sorting
- need fast
- need server
and thinking: how is not this a readonly slave database? I think that
mysql can either do this with a readonly slave *or* it cannot be done
with modest resources.

The problem is that for a perfectly normalized database, those queries
are *heavy*.

We're using straight, direct SQL (no ActiveRecord calls) there, and
several DBAs have already looked into our query strategy. Bottom line
is that each query on the normalized database is non-trivial, and they
can not reduce it to less than 0.2 secs / query. As we've 5+ of these
queries per page, we'd need one MySQL server for every
request-per-second we want to serve. As we need at least 50 reqs/sec,
we'd need 50 MySQL servers (and probably something similar in terms of
web servers). We can not afford that.

We can only improve the queries TTC by replicating data inside the
database, i.e. de-normalizing it with internal caching at the table

<database> A collection of {records} in a {relational database}.

	(1997-06-04)

level (basically, that amounts to replicating certain columns from
table `bars` in table `foo

<jargon> /foo/ A sample name for absolutely anything,
	especially programs and files (especially {scratch files}).
	First on the standard list of {metasyntactic variables} used
	in {syntax} examples.  See also {bar}, {baz}, {qux}, {quux},
	{corge}, {grault}, {garply}, {waldo}, {fred}, {plugh},
	{xyzzy}, {thud}.

	The etymology of "foo" is obscure.  When used in connection
	with "bar" it is generally traced to the WWII-era Army slang
	acronym {FUBAR}, later bowdlerised to {foobar}.

	However, the use of the word "foo" itself has more complicated
	antecedents, including a long history in comic strips and
	cartoons.

	"FOO" often appeared in the "Smokey Stover" comic strip by
	Bill Holman.  This surrealist strip about a fireman appeared
	in various American comics including "Everybody's" between
	about 1930 and 1952.  FOO was often included on licence plates
	of cars and in nonsense sayings in the background of some
	frames such as "He who foos last foos best" or "Many smoke but
	foo men chew".

	Allegedly, "FOO" and "BAR" also occurred in Walt Kelly's
	"Pogo" strips.  In the 1938 cartoon "The Daffy Doc", a very
	early version of Daffy Duck holds up a sign saying "SILENCE IS
	FOO!".  Oddly, this seems to refer to some approving or
	positive affirmative use of foo.  It has been suggested that
	this might be related to the Chinese word "fu" (sometimes
	transliterated "foo"), which can mean "happiness" when spoken
	with the proper tone (the lion-dog guardians flanking the
	steps of many Chinese restaurants are properly called "fu
	dogs").

	Earlier versions of this entry suggested the possibility that
	hacker usage actually sprang from "FOO, Lampoons and Parody",
	the title of a comic book first issued in September 1958, a
	joint project of Charles and Robert Crumb.  Though Robert
	Crumb (then in his mid-teens) later became one of the most
	important and influential artists in underground comics, this
	venture was hardly a success; indeed, the brothers later
	burned most of the existing copies in disgust.  The title FOO
	was featured in large letters on the front cover.  However,
	very few copies of this comic actually circulated, and
	students of Crumb's "oeuvre" have established that this title
	was a reference to the earlier Smokey Stover comics.

	An old-time member reports that in the 1959 "Dictionary of the
	TMRC Language", compiled at {TMRC} there was an entry that
	went something like this:

	FOO: The first syllable of the sacred chant phrase "FOO MANE
	PADME HUM."  Our first obligation is to keep the foo counters
	turning.

	For more about the legendary foo counters, see {TMRC}.  Almost
	the entire staff of what became the {MIT} {AI LAB} was
	involved with TMRC, and probably picked the word up there.

	Another correspondant cites the nautical construction
	"foo-foo" (or "poo-poo"), used to refer to something
	effeminate or some technical thing whose name has been
	forgotten, e.g. "foo-foo box", "foo-foo valve".  This was
	common on ships by the early nineteenth century.

	Very probably, hackish "foo" had no single origin and derives
	through all these channels from Yiddish "feh" and/or English
	"fooey".

	[{Jargon File}]

	(1998-04-16)

s`, thus saving some very heavy JOINs).

But if we're already de-normalizing, caching and replicating data, we
might as well create another layer of de-normalized, processed data
between the database and the Rails servers. That way, we'll need
less MySQL servers, output

<architecture> {Data} transferred from a computer system to
	the outside world via some kind of {output device}.

	Opposite: {input}.

	(1997-04-28)

requests faster (as the layer'd hold
the data in an already processed state), and save a much of the
replication / clustering

<file system> An elementary unit of allocation of a disk made
	up of one or more physical {blocks}.

	A {file} is made up of a whole number of possibly
	non-contiguous clusters.  The cluster size is a tradeoff
	between space efficiency (the bigger is the cluster, the
	bigger is on the average the wasted space at the end of each
	file) and the length of the {FAT}.

	(1996-11-04)

overhead

1. Resources (in computing usually processing time or storage
	space) consumed for purposes which are incidental to, but
	necessary to, the main one.  Overheads are usually
	quantifiable "costs" of some kind.

	Examples: The overheads in running a business include the cost
	of heating the building.  Keeping a program running all the
	time eliminates the overhead of loading and initialising it
	for each transaction.  Turning a {subroutine} into {inline}
	code eliminates the call and return time overhead for each
	execution but introduces space overheads.

	2. <communications> information, such as control, routing, and
	error checking characters, that is transmitted along with the
	user data.  It also includes information such as network
	status or operational instructions, network routing
	information, and retransmissions of user data received in
	error.

	3. Overhead transparencies or "slides" (usually 8-1/2" x 11")
	that are projected to an audience via an overhead (flatbed)
	projector.

	(1997-09-01)

.

-Tom

From: Tom Machinski

Date: Sunday, October 28, 2007

wrote in message:

I do not recommend you use this project, I have not used it myself for quite a
while and it has a number of issues I have not address

1. <networking> {e-mail address}.

	2. <networking> {Internet address}.

	3. <networking> {MAC address}.

	4. <storage, programming> An unsigned integer used to select
	one fundamental element of storage, usually known as a {word}
	from a computer's {main memory} or other storage device.  The
	{CPU} outputs addresses on its {address bus} which may be
	connected to an {address decoder}, {cache controller}, {memory
	management unit}, and other devices.

	While from a hardware point of view an address is indeed an
	integer most {strongly typed} programming languages disallow
	mixing integers and addresses, and indeed addresses of
	different data types.  This is a fine example for {syntactic
	salt}: the compiler could work without it but makes writing
	bad programs more difficult.

	(1997-07-01)

ed. You may find it a
helpful basis implementation if you should decided to go the pure Ruby
route.

Thanks, Ian!

Would you mind sharing - here, or by linking a blog / text, or
privately if you prefer - some information about these issues?

I'm asking for two reasons:

1) To learn about possible pitfalls / complications / costs involved
in taking the pure Ruby route.

2) We may decide to adopt your project and try to address those issues
to use the patched Boogaloo in production.

Thanks,
-Tom

From: Andreas S.

Date: Sunday, October 28, 2007

wrote in message:
wrote in message:
with modest resources.
The problem is that for a perfectly normalized database, those queries
are *heavy*.
We're using straight, direct SQL (no ActiveRecord calls) there, and
several DBAs have already looked into our query strategy. Bottom line
is that each query on the normalized database is non-trivial, and they
can not reduce it to less than 0.2 secs / query.

Try enabling

<programming> (Or "enabling") Modification of the design or
	implementation of software to allow {internationalisation} to
	take place.

	In particular, enabling may refer to the modification of
	software to support double-byte character sets, hence
	"{Unicode} enabling" and "double-byte enabling".

	(1999-06-28)

the MySQL query cache. For many applications even a few MB
can work wonders.
--
Posted via http://www.ruby-forum.com/.

From: Tom Machinski

Date: Sunday, October 28, 2007

wrote in message:

Try enabling the MySQL query cache. For many applications even a few MB
can work wonders.

Thanks, that's true, and we already do that. We've a very large
cache in fact (~500 MB) and it does improve performance, though not
enough.

-Tom

From: Stanislav Sedov

Date: Sunday, October 28, 2007

On Sun, Oct 28, 2007 at 07:31:30AM +0900 Tom Machinski mentioned:

2) Is this task doable in Ruby? Or maybe only a Ruby + X combination
(X probably being C)?

I beleive, you can achive a high efficiency in server design by using
event

1. <software> An occurrence or happening of significance to a
	task or program, such as the completion of an asynchronous
	input/output operation.  A task may wait for an event or any
	of a set of events or it may (request to) receive asynchronous
	notification (a {signal} or {interrupt}) that the event has
	occurred.

	See also {event-driven}.

	2. <data> A transaction or other activity that affects the
	records in a file.

	(2000-02-09)

-driven design. There're some event libraries for ruby available,
e.g. eventmachine. In this case the scalability

How well a solution to some problem will work when the size of
	the problem increases.

	For example, a central {server} of some kind with ten
	{clients} may perform adequately but with a thousand clients
	it might fail to meet response time requirements.  In this
	case, the average response time probably scales linearly with
	the number of clients, we say it has a {complexity} of O(N)
	("order N") but there are problems with other complexities.
	E.g. if we want N nodes in a network to be able to communicate
	with each other, we could connect each one to a central
	exchange, requiring O(N) wires or we could provide a direct
	connection between each pair, requiring O(N^2) wires (the
	exact number or formula is not usually so important as the
	highest power of N involved).

	(1995-03-29)

of the server should
be comparable with the C version.

Thread will have a huge overhead in case of many clients.

BTW, the original memcached uses event-driven design too, IIRC.

--
Stanislav Sedov
ST4096-RIP

<audio> (From "rip off" - to steal) To copy audio or video
	(e.g. from a {compact disc}) to a {file}, e.g. an {MP3}
	{digital audio} file, on a {computer} {hard disk}.

	While it may be legal to do this for personal use, ripping a
	copyright work and distributing the result to others could
	result in prosecution.

	(2006-01-27)

From: Tom Machinski

Date: Sunday, October 28, 2007

wrote in message:

I beleive, you can achive a high efficiency in server design by using
event-driven design. There're some event libraries for ruby available,
e.g. eventmachine. In this case the scalability of the server should
be comparable with the C version.
Thread will have a huge overhead in case of many clients.
BTW, the original memcached uses event-driven design too, IIRC.

Yes, memcached (including latest) uses libevent.

I'm not completely sure whether a production-grade server of this sort
is feasible in Ruby. Many people, both here and elsewhere, seem to
think it should be done in C for better stability / efficiency /
resource consumption.

Thanks,
-Tom

--
Stanislav Sedov
ST4096-RIPE

From: Tom Machinski

Date: Sunday, October 28, 2007

wrote in message:

Add "large set of very large (binary?) objects". So ... yes, at least
*one* database/server. This is exactly the sort of thing you *can* throw
hardware at. I guess I'd pick PostgreSQL over MySQL for something like
that, but unless you're a billionaire, I'd be doing it from disk and not
from RAM. RAM-based "databases" look really attractive on paper, but
they tend to look better than they really are for a lot of reasons:
1. *Good* RAM -- the kind that does not fall over in a ragged heap when
challenged with "memtest86" -- isn't inexpensive. Let's say the objects
are "very large" -- how about a typical CD length of 700 MB? OK ... too
big -- how about a three minute video highly compressed. How big are
those puppies? Let's assume a megabyte. 100K of those is 100 GB. Wanna
price 100 GB of *good* RAM? Even with compression, it does not take much
stuff to fill up a 160 GB iPod, right?

I might've impressed you with a somewhat inflated view of how large
our data-set is :-)

We've about 100K objects, occupying ~500KB per object. So all in
all, the total weight of our dataset is no more than 500MBs. We might
grow to maybe twice that in the next 2 years. But that's it.

So it's very feasible to keep the entire data-set in *good* RAM for a
reasonable cost.

2. A good RDBMS design / query planner is amazingly intelligent, and you
can give it hints. It might take you a couple of weeks to build your
indexes but your queries will run fast afterwards.

Good point. Unfortunately, MySQL 5 does not appear to be able to take
hints. We have analyzed our queries and there's some strategies there we
could definitely improve by manual hinting, but alas we'd need to
switch to an RDBMS that supports those.

3. RAID 10 is your friend. Mirroring preserves your data when a disk
dies, and striping makes it come into RAM quickly.
4. Enterprise-grade SANs have lots of buffering built in. And for that
stuff, you do not have to be a billionaire -- just a plain old millionaire.

We had some bad experience with a poor SAN setup, though we might've
been victims of improper installation.

Thanks,
-Tom

From: Tom Machinski

Date: Sunday, October 28, 2007

wrote in message:

So alter memcached to accept

<library, networking> {Berkeley} {Unix} networking {socket}
	library routine to satisfy a connection request from a remote
	{host}.  A specified socket on the local host (which must be
	capable of accepting the connection) is connected to the
	requesting socket on the remote host.  The remote socket's
	socket address is returned.

	{Unix manual pages}: accept(2), connect(2).

	(1994-11-08)

a 'query' in the form of arbitrary ruby (or
perhaps a pre-defined ruby) that a peer-daemon is to execute

<operating system, programming> The process of carrying out
	the {instructions} in a computer program by a computer.

	See also {dry run}.

	(1996-05-13)

over the set of
results a particular memcached node

1. A point or vertex in a {graph}.

	2. {network node}.

	3. A {hypertext} document.

contains.

Yeah, I thought of writing a Ruby daemon that "wraps" memcached.

But then the wrapper

<programming> Code which is combined with another piece of
	{code} to determine how that code is executed.  The wrapper
	acts as an interface between its caller and the wrapped code.
	This may be done for compatibility, e.g. if the wrapped code
	is in a different programming language or uses different
	calling conventions, or for security, e.g. to prevent the
	calling program from executing certain functions.  The
	implication is that the wrapped code can only be accessed via
	the wrapper.

	(1998-12-15)

would've to deal with all the performance
challenges that a full replacement to memcached has to deal with,
namely: handling multiple concurrent clients, multiple simultaneous
read/write requests
(race conditions etc.) and heavy loads.

A naive

Untutored in the perversities of some particular program or
	system; one who still tries to do things in an intuitive way,
	rather than the right way (in really good designs these
	coincide, but most designs aren't "really good" in the
	appropriate sense).  This trait is completely unrelated to
	general maturity or competence or even competence at any other
	specific program.  It is a sad commentary on the primitive
	state of computing that the natural opposite of this term is
	often claimed to be "experienced user" but is really more like
	"cynical user".

	(1994-11-29)

implementation of memcached itself'd be trivial to write;
memcached's real merits aren't its rather limited featureset, but
its performance, stability, and robustness - i.e., its capability

<operating system, security> An {operating system} security or
	access control model where specific types of access to a
	specific object are granted by giving a process this data
	structure or {token}.

	The token may be unforgeable (typically by using {encryption}
	or hardware "tagged" memory).  Capabilities are used in OSes
	such as {Hydra}, {KeyKOS}, {EROS}, {Chorus}/{Mix}, and the
	{Stanford V system}.  Similar to {Kerberos}, but in an OS
	context.

	Compare {access control list}.

	(1998-03-08)

to
overcome the above challenges.

The only way I could use memcached to do complex queries is by
patching memcached to accept and handle

1. <programming, operating system> A simple item of data that
	identifies a resource.  For example, a {Unix} file handle
	identifies an open file and associated data such as whether it
	was opened for read or write and the current read/write
	position.  On the {Macintosh}, a handle is a pointer to a
	pointer to some dynamically-allocated memory.  The extra level
	of indirection allows on-the-fly {memory compaction} or
	{garbage collection} without invalidating application program
	references to the allocated memory.

	2. <jargon> An alias used intended to conceal a user's true
	identity in an electronic message.  The term is common on
	Citizen's Band and other amateur radio but, in that context
	usually means the user's real name as {FCC} rules forbid
	concealing one's identity.

	Use of grandiose handles is characteristic of {crackers},
	{weenies}, {spods}, and other lower forms of network life;
	true hackers travel on their own reputations.

	Compare {nick}.

	[{Jargon File}]

	(2004-07-20)

complex queries. Such a patch
won't have anything to do with Ruby itself,'d probably be very
non-trivial, and will have to significantly extend

<programming> To add {features} to a program, especially
	through the use of {hooks}.

	"Extend" is very often used in the phrase "extend the
	{functionality} of a program."

	{Plug-ins} are one form of extension.

	(1997-06-21)

memcached's
architecture. I doubt I have the time to roll out something like that.

-Tom

From: marc spitzer

Date: Sunday, October 28, 2007

wrote in message:
wrote in message:
The other thing you can play with is using sqlite as the local (one
per app server) cache engine.
Thanks, but if I'm already caching at the local process level, I might
as well cache to in-memory Ruby objects; the entire data-set is not
that huge for a high-end server RAM capacity: about 500 MB all in all.

What'd happen if you used two stages of mysql databases? What I mean
is that you've your production db with all your nice clean structure for
writing new data to and as a master

<chat> The owner of a {bot}.

	(1997-04-07)

source

<language, programming> (Or "source", or rarely "source
	language") The form in which a computer program is written by
	the programmer.  Source code is written in some formal
	programming language which can be compiled automatically into
	{object code} or {machine code} or executed by an
	{interpreter}.

	(1995-01-05)

for your horrible demoralized
db then you've a job that pushes changes every N minutes to the evil
ugly read only db. It is a new step in production, but it does allow you
to stick with the same tech mix you are using now.

marc
--
ms4720@xxxxxxxxxxx
SDF Public Access

1. <language> An English-like query language used in the
	{Pick} {operating system}.

	2. <database, product> {Microsoft Access}.

	(1994-11-08)

UNIX System - http://sdf.lonestar.org

Next Message: [ANN] RMagick 2.0.0 beta5 builds with Ruby 1.9.0