断点续传
通常视频文件都比较大，所以对于媒资系统上传文件的需求要满足大文件的上传要求。http协议本身对上传文件大小没有限制，但是客户的网络环境质量、电脑硬件环境等参差不齐，如果一个大文件快上传完了网断了没有上传完成，需要客户重新上传，用户体验非常差，所以对于大文件上传的要求最基本的是断点续传。
什么是断点续传：
引用百度百科：断点续传指的是在下载或上传时，将下载或上传任务（一个文件或一个压縮包）人为的划分为几个部分，每一个部分采用一个线程进行上传或下载，如果碰到网络故障，可以从已经上传或下载的部分开始继续上传下载末完成的部分，而没有必要从头开始年传下载，断点续传可以提高节省操作时间，提高用户体验性。
业务流程
1、前端会发起checkFile请求，后端检查文件是否在数据库和MinIO中，如果不在直接返回false，在数据库中但不在MinIO中也返回false。ID为文件MD5。
2、文件不存在，前端将文件分块；分块上传前会再次检查分片是否在MinIO中，如果不存在返回false。
3、分块不存在，前端发起分块上传请求，后端将文件转存至MinIO中；分块存在，前端继续重复2-3步骤，直至所有分块上传完成。
4、分块上传完成后，前端发起合并文件请求，后端向MinIO发起合并请求；合并完成后，将数据保存到数据库中。
5、后端向MinIO中发起删除分块请求。
接口信息
文件上传前的检查文件
分块文件上传前的检查文件
上传分块文件
合并分块文件
定义Service
文件存储操作
定义上传媒体文件的方法，找到com.swx.media.service包下的FileStorageService接口，定义如下方法：
FileStorageService
public String uploadChunkFile(String path, String mimeType, InputStream inputStream);
public void mergeFile(String folder, String filepath, int chunkSize) throws Exception;
public ObjectStat getObjectStat(String bucket, String filepath);
void clearChunkFiles(String chunkFolder, int chunkTotal);
|
实现该方法
MinIOFileStorageServiceprivate final MinioClient minioClient;
private final MinIOConfigProperties minIOConfigProperties;
private final static String separator = "/";
public MinIOFileStorageService(MinioClient minioClient, MinIOConfigProperties minIOConfigProperties) {
this.minioClient = minioClient;
this.minIOConfigProperties = minIOConfigProperties;
}
@Override
public String uploadChunkFile(String path, String mimeType, InputStream inputStream) {
try {
PutObjectArgs putObjectArgs = PutObjectArgs.builder()
.object(path)
.bucket(minIOConfigProperties.getBucket().get("video"))
.stream(inputStream, inputStream.available(), -1)
.contentType(mimeType)
.build();
minioClient.putObject(putObjectArgs);
return path;
} catch (Exception ex) {
log.error("minio put file error.", ex);
throw new RuntimeException("上传文件失败");
}
}
@Override
public void mergeFile(String folder, String filepath, int chunkSize) throws Exception {
try {
String bucket = minIOConfigProperties.getBucket().get("video");
List<ComposeSource> sources = Stream.iterate(0, i -> ++i).limit(chunkSize)
.map(i -> ComposeSource.builder().bucket(bucket).object(folder + i).build()).collect(Collectors.toList());
ComposeObjectArgs composeObjectArgs = ComposeObjectArgs.builder()
.object(filepath)
.bucket(bucket)
.sources(sources)
.build();
minioClient.composeObject(composeObjectArgs);
} catch (Exception ex) {
log.error("minio compose file error.", ex);
throw new Exception("合并文件失败");
}
}
@Override
public ObjectStat getObjectStat(String bucket, String filepath) {
try {
return minioClient.statObject(
StatObjectArgs.builder().bucket("video").object(filepath).build());
} catch (Exception e) {
log.error("文件信息获取失败, 桶: {}, 文件路径: {}", bucket, filepath, e);
return null;
}
}
@Override
@Async
public void clearChunkFiles(String chunkFolder, int chunkTotal) {
List<DeleteObject> objects = Stream.iterate(0, i -> ++i).limit(chunkTotal)
.map(i -> new DeleteObject(chunkFolder.concat(Integer.toString(i)))).collect(Collectors.toList());
RemoveObjectsArgs removeObjectsArgs = RemoveObjectsArgs.builder()
.bucket(minIOConfigProperties.getBucket().get("video"))
.objects(objects)
.build();
Iterable<Result<DeleteError>> results = minioClient.removeObjects(removeObjectsArgs);
for (Result<DeleteError> result : results) {
DeleteError error = null;
try {
error = result.get();
log.info("Error in deleting object {}; {}", error.objectName(), error.message());
} catch (Exception e) {
log.error("清理文件出错", e);
}
}
}
|
文件上传方法
定义上述方法，找到com.swx.media.service包下的MediaFilesService接口，定义如下方法：
MediaFilesService
Boolean checkFile(String fileMd5);
Boolean checkChunk(String fileMd5, int chunk);
Boolean uploadChunk(MultipartFile file, String fileMd5, int chunk);
R mergeChunks(Long companyId, String fileMd5, int chunkTotal, UploadFileParamDTO dto);
|
实现该方法
MediaFilesServiceImplprivate final FileStorageService fileStorageService;
private final TransactionTemplate transactionTemplate;
private final MinIOConfigProperties minIOConfigProperties;
public MediaFilesServiceImpl(FileStorageService fileStorageService, TransactionTemplate transactionTemplate, MinIOConfigProperties minIOConfigProperties) {
this.fileStorageService = fileStorageService;
this.transactionTemplate = transactionTemplate;
this.minIOConfigProperties = minIOConfigProperties;
}
@Override
public Boolean checkFile(String fileMd5) {
MediaFiles dbMediaFile = getById(fileMd5);
if (dbMediaFile != null) {
ObjectStat objectStat = fileStorageService.getObjectStat(dbMediaFile.getBucket(), dbMediaFile.getFilePath());
return objectStat != null;
}
return false;
}
@Override
public Boolean checkChunk(String fileMd5, int chunk) {
String path = getChunkFileFolderPath(fileMd5) + chunk;
String bucket = minIOConfigProperties.getBucket().get("video");
ObjectStat objectStat = fileStorageService.getObjectStat(bucket, path);
return objectStat != null;
}
@Override
public Boolean uploadChunk(MultipartFile file, String fileMd5, int chunk) {
try {
String mineType = getMineType(null);
String chunkFilePath = getChunkFileFolderPath(fileMd5) + chunk;
fileStorageService.uploadChunkFile(chunkFilePath, mineType, file.getInputStream());
return true;
} catch (IOException e) {
log.error("分块文件上传失败，文件ID：{}，分块序号: {}", fileMd5, chunk, e);
return false;
}
}
@Override
public R mergeChunks(Long companyId, String fileMd5, int chunkTotal, UploadFileParamDTO dto) {
String chunkFileFolderPath = getChunkFileFolderPath(fileMd5);
String suffix = dto.getFilename().substring(dto.getFilename().lastIndexOf("."));
String filepath = getFilePathByMd5(fileMd5, suffix);
String bucket = minIOConfigProperties.getBucket().get("video");
try {
fileStorageService.mergeFile(chunkFileFolderPath, filepath, chunkTotal);
} catch (Exception e) {
log.error("文件合并失败，文件名: {}", filepath, e);
return R.fail(false, "文件合并失败");
}
ObjectStat objectStat = fileStorageService.getObjectStat(bucket, filepath);
if (objectStat == null) {
return R.fail(false, "合并文件获取失败");
}
dto.setFileSize(objectStat.length());
MediaFiles mediaFiles = transactionTemplate.execute(transactionStatus -> {
return saveAfterStore(dto, companyId, fileMd5, bucket, filepath);
});
if (mediaFiles == null) {
return R.fail(false, "文件保存失败");
}
fileStorageService.clearChunkFiles(chunkFileFolderPath, chunkTotal);
return R.success(true);
}
private String getMineType(String suffix) {
if (suffix == null) suffix = "";
ContentInfo extensionMatch = ContentInfoUtil.findExtensionMatch(suffix);
String mimeType = MediaType.APPLICATION_OCTET_STREAM_VALUE;
if (extensionMatch != null) {
mimeType = extensionMatch.getMimeType();
}
return mimeType;
}
private String getFileMd5(InputStream inputStream) {
try {
return DigestUtils.md5DigestAsHex(inputStream);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
private String getFilePathByMd5(String fileMd5, String suffix) {
return fileMd5.charAt(0) + "/" + fileMd5.charAt(1) + "/" + fileMd5 + "/" + fileMd5 + suffix;
}
private String getChunkFileFolderPath(String fileMd5) {
return fileMd5.charAt(0) + "/" + fileMd5.charAt(1) + "/" + fileMd5 + "/chunk/";
}
|
定义Controller
MediaFilesUploadControllerprivate final MediaFilesService mediaFilesService;
public MediaFilesUploadController(MediaFilesService mediaFilesService) {
this.mediaFilesService = mediaFilesService;
}
@ApiOperation("文件上传前的检查文件")
@PostMapping("/checkfile")
public Boolean checkFile(@RequestParam("fileMd5") String fileMd5) {
return mediaFilesService.checkFile(fileMd5);
}
@ApiOperation("分块文件上传前的检查文件")
@PostMapping("/checkchunk")
public Boolean checkChunk(@RequestParam("fileMd5") String fileMd5,
@RequestParam("chunk") int chunk) {
return mediaFilesService.checkChunk(fileMd5, chunk);
}
@ApiOperation("上传分块文件")
@PostMapping("/uploadchunk")
public Boolean uploadChunk(@RequestParam("file") MultipartFile file,
@RequestParam("fileMd5") String fileMd5,
@RequestParam("chunk") int chunk) {
return mediaFilesService.uploadChunk(file, fileMd5, chunk);
}
@ApiOperation("合并文件")
@PostMapping("/mergechunks")
public R mergeChunks(@RequestParam("fileName") String fileName,
@RequestParam("fileMd5") String fileMd5,
@RequestParam("chunkTotal") int chunkTotal) {
Long companyId = 1232141425L;
UploadFileParamDTO dto = new UploadFileParamDTO();
dto.setFilename(fileName);
dto.setFileType("001002");
dto.setTags("视频文件");
return mediaFilesService.mergeChunks(companyId, fileMd5, chunkTotal, dto);
}
|
Push到Git
