What you need to know about object storage

Storage architectures from the early days of computing typically used block storage and managed data as hierarchy of files. While this was fine when the amount of data involved was relatively small, as storage requirements grew it meant that the management of data became a major task. What was needed was a way of making handling data simpler whilst allowing scalability and ease of access to the information.

Object storage treats data as objects, each being made up of the data itself, metadata that defines it and a unique identifier. The advantage of this is that it makes it possible to store large amounts of unstructured data in a way that’s affordable, scalable and capable of a degree of self-management. In recent years its use has been driven by online sharing and streaming services, allowing a diverse range of digital content to be easily and quickly accessed from anywhere without the need for complex, specialist software.

How It Works

The concept of object storage dates back to the mid 1990s, but the earliest commercial product was the EMC Centera system launched in 2003.

Object storage systems work by separating the lower storage layers from the application layers. This allows the data to be handled as objects rather than files or blocks. Because each object carries with it its own administrative and descriptive information it cuts the need for administrators to carry out storage management functions like dividing the disk into logical volumes or using RAID to deal with failures.

Because metadata is linked to each object it can be used for a number of different purposes. For example application or user specific information about the object can be captured and used to improve indexing. The metadata can also be used in setting data management policies and centralizing control of storage.

Object oriented storage devices can use low-cost commodity hardware and use an interface that allows for the creation, deletion and modifying of objects. The software also allows for security controls to be applied at the object level. Interfaces at a program level using APIs allow applications to access and manipulate data objects directly without the need for intermediate layers of file system.

Operational Standards

As with other key areas of information technology, object storage relies on having commonly accepted standards to ensure that systems are interoperable. The first Object-based Storage Device standard, OSD-1, specified 64-bit object and partition IDs. Although objects are created and deleted within partitions, neither has a fixed size and there are no physical size limitations. Attributes like the size of an object, when it was modified, and a policy tag are read by the OSD interface, but there are also other attributes which are used by higher level systems.

The second generation OSD-2 standard added extra support for collection objects, improved error handling, and the ability to take a read-only ‘snapshot’ copy of all the objects in a partition. Collection objects contain the identifiers of other objects, these can be used for error reporting and correction by recording media failure or software errors to an error collection that can be used by higher level storage to take corrective action.

Uses

The advantages of object storage have led to its being introduced in a number of different areas of business. Early adopters of the technology were those companies needing archive storage for large volumes of data. The scalable nature of object storage means it’s easier to expand as the volume of the data grows. EMC Centera had around 3,500 customers by 2007. One of the first object storage file systems, Lustre, is used in many of the world’s most powerful supercomputers.

Object storage is also widely used by cloud storage providers. Services like Dropbox, Microsoft Azure and Google Cloud Storage all make use of it to allow the storage and exchange of data. These services deal with huge volumes of files. In 2014 Azure says it was storing around 20 trillion objects.

Facebook implemented its own object storage system known as Haystack to allow it to efficiently store and manage large volumes of photos. The system allows Facebook to store more than 240 billion photos and adding to them at the rate of around 350 million a day.

The Object Storage Market

Research specialist IDC identified 18 active players in the object storage market in 2014. The same report predicts that by 2018 the market will have grown by 27 percent over its 2014 level.

The main players have tended to be companies that specialize in storage rather than general IT providers. Among the market leaders are Scality with its Scality Ring product. This can run on commodity hardware and is aimed at service providers and broadcasters who need to deal with large amounts of unstructured data such as emails and media files.

IBM staked its claim in the sector with its takeover of Cleversafe which has been rebranded as IBM Cloud Object Storage. It’s designed to work with in-house, cloud and hybrid storage solutions, and offers straightforward management and security options.

Where Next?

Object storage has already made its mark with cloud service providers and media companies which have to deal with very large quantities of data. But what about other businesses? Technology is allowing companies to collect more and more information about their customers. This comes not only from transaction records, but also increasingly from Internet of Things devices.

Coping with this influx of big data and being able to make sense of it means that demands on storage will increase significantly before the end of the decade. Predictions vary but enterprises could be storing 30 times the amount of data they do today. This is likely to drive the adoption of object storage into more and more companies as they seek solutions to store and access the information they collect.

Moving to object storage makes scalability easier and allows storage to beyond petabyte level if required. Crucially the storage is treated as a single system whatever its size. But, perhaps even more important, object storage reduces the workload required to manage large amounts of storage storage. Without it, holding the quantities of data that are going to be required in future will become impractical.

Published under license from ITProPortal.com, a Future plc Publication. All rights reserved.

Photo Credit: wavebreakmedia/Shutterstock