Sleep is an essential phenomenon for maintaining good health and wellbeing. According to several existing studies, sleep is often disturbed by depression, anxiety, or fear. In this work, we perform a depression-specific imaging genetic association study to examine the relationship between depression-specific imaging genetic biomarkers and quality of sleep. Specifically, joint-connectivity-based sparse canonical correlation analysis (JCB-SCCA) was applied to identify biomarkers between multi-model imaging scans (i.e., functional magnetic resonance imaging [fMRI] and diffusion tensor imaging [DTI]) and genotype data (i.e., single nucleotide polymorphism, SNP). Using the identified imaging and genetic biomarkers, we further built a regression model for predicting the Pittsburgh sleep quality index. The proposed model was compared with five different models using different sets of biomarkers. Our empirical study was performed on the Human Connectome Project (HCP) data, and demonstrated that the JCB-SCCA model discovered stronger imaging genetic associations than the traditional SCCA method. Furthermore, the imaging and genetic biomarkers identified from our association analysis yielded improved performance over those from the other models in predicting the quality of sleep.