How to break the large-scale cluster management problem of security cloud storage?

Jiang Wenlong, from the Dahua Research Institute of Zhejiang Dahua Technology Co., Ltd., said: For the business characteristics of the security industry, the following two major technical directions for cloud storage need to be broken:

1. Efficient metadata organization and framework construction to solve the problem of large-scale cluster management and massive files.

The number of nodes that need to be managed in the entire distributed system is hundreds or thousands. A real file of the user will be distributed on multiple nodes, and multiple nodes are responsible for carrying the write of real data. When reading, it is necessary to request the data location information through the metadata management server to initiate reading. The performance of the metadata request is a step-by-step recursion or a single access to complete the operation, is a key factor to measure the performance of the entire system.

For a single large file, whether it can fully exert the read and write performance involves the problem of split granularity. As the core of the metadata service, it is required to support high-speed concurrent processing in thousands of nodes and tens of thousands of clients. This needs to be considered in the basic protocol framework and signaling interaction model, and serialized by ultra-high protocols. De-serialization performance, scalable protocol design, network framework model, and task processing model are all layered up and down, and are processed efficiently in each link. A reasonable number of organizational structures can use the bucket-based method of type object storage, so that the data hash is distributed, and the file is simple and efficient to manage. For the data in the bucket, it is not necessary to adopt a traditional directory tree to perform step-by-step traversal, only once. Positioning can complete the operation.

For the data block organization management of files, on the one hand, it is necessary to control the better granularity to achieve IO. The advantages of multi-node and multi-disk can be fully utilized. On the other hand, it is necessary to reduce the management pressure of metadata and improve the number of clusters and the number of files managed. The user's data block exists on the storage node and is divided into segments in each disk. The system runs for a long time or restarts, powers down, byte jumps, etc., and needs to be able to perform data blocks in the node management and data blocks in the metadata. Compare and find out the difference item to complete the correction, and trigger the recovery of the damaged data early. This requires the metadata to be reasonably organized, and can quickly find the metadata information of the corresponding node, and does not affect other metadata in real time during the comparison process. Access and add.

2. A clear read-write model provides business usage semantics and addresses different write and read requirements for video and images.

Common read and write does not provide an interface, and requires explicit read and write semantics. For example, the file system provides file operation semantics, according to the open/write/read/close mode, and supports the semantics of seek and modification and appendment; the S3 interface provides the putObject/getObject interface, which can be seen after uploading once. Semantics; HDFS provides operational semantics similar to file systems, but does not support modification.

For video, it should follow the semantics of the file but does not need to support append and modify. It only needs to support streaming write, and supports reading while reading, to avoid the business layer needs to open a large cache or cache the video file locally to upload. The same is true for the image writing method, and file stream mode writing should also be supported. Although it seems that the picture can be written at one time, but the current picture can be 1MB or larger, and only by setting the cache size to complete a picture write of the application, there will be memory usage in the cloud storage client. Too large or not smooth enough to write will have a meal effect and cause the cache to be full. On the read, there is no need to read the data in one picture, but the whole write is completed immediately.

From the perspective of the file name, since each picture corresponds to a front-end capture record, the picture address can be stored along with the structured record, and the user does not need to be related to the picture address generation mode, which means that the picture address can be returned by the system. Generate. For the video file formed after the video stream is stored, the user can record the file name by using the specified file name capability provided by the cloud storage, and generate the file name according to the customized business logic, and then query according to the rules. Complete the recording list or specify the playback of the recording file.